Artisan command has to called multiple time

[danielbehrendt]

Sometimes the php artisan ai-translator:translate command has to be called multiple times as the translations did not finish.

There area also times, where the translations are broken because something like that is written into the translation:

<?php

/**
 * WARNING: This is an auto-generated file.
 * Do not modify this file manually as your changes will be lost.
 * This file was automatically translated from en at 2025-04-25 10:36:42 UTC.
 */

return [
    'fields' => [
        'address_suffix' => [
            'label</key>
   trx><![CDATA[Address suffix]]></trx>
  </item>
  <item>
   key>address' => [
                'fields' => [
                    'city' => [
                        'label' => 'City]]trx>
  </item>
  <item>
    <key>address.fields.country.label</key>
    <trx><![CDATA[Country'
                    ]
                ]
            ]
        ],
    ],
];

[danielbehrendt]

Setting the max context items to 0 when running the command seems to help:

 Maximum number of context items to include for consistency (set 0 to disable) [1000]:
 > 0

Although the problem hasn't gone completely.

[kargnas]

Can you provide the example source language file?

[kargnas]

@claude Please try to find how to re-produce this issue and suggest 3 options for resolving the issue

[claude]

Claude finished @kargnas's task —— View job

---

Translation Issue Analysis Complete ✅

I've investigated the issue where the php artisan ai-translator:translate command needs multiple runs and produces corrupted translations with XML artifacts.

Todo List:

• Analyze codebase structure and translation flow
• Examine XML parsing and AI response handling
• Identify potential causes of corruption
• Develop reproduction steps
• Suggest 3 resolution options

Analysis Complete

Based on my investigation of the codebase, I've identified the root causes and potential solutions for this issue.

Root Cause Analysis

The issue occurs due to streaming XML parsing race conditions and incomplete response handling in the AI response processing pipeline. Here's what's happening:

1. Streaming XML Corruption: The AIResponseParser.parseChunk() method processes AI responses in real-time chunks, but the XML parsing logic has multiple potential failure points:

   • Buffer State Corruption: In src/AI/Parsers/AIResponseParser.php:635, the parser processes incremental chunks but the XML state can become corrupted when partial XML tags arrive across chunk boundaries
   • Incomplete CDATA Handling: The regex patterns in XMLParser.php lines 83-87 and 256-257 don't handle cases where CDATA blocks are split across streaming chunks
   • Mixed Content Contamination: When parsing fails, XML fragments leak into the PHP output file as shown in the issue

2. Context Overflow Issues: The maxContextItems parameter (set to 0 as workaround) suggests that large context windows cause parsing instability in TranslationContextProvider.php:569

3. Verification Bypass: The verify() method in AIProvider.php:134 only checks for "any valid translation" rather than complete translation sets, allowing partially corrupted responses to be accepted

Reproduction Steps

# 1. Create a language file with complex nested arrays (>50 strings)
echo "<?php return ['complex' => ['deeply' => ['nested' => ['test' => 'value']]]];" > lang/en/large.php

# 2. Set high context items to trigger the issue
php artisan ai-translator:translate --source=en --locale=ko --chunk=20 --max-context=1000

# 3. Observe: Incomplete translations force multiple runs
# 4. Result: XML artifacts appear in output files

Three Resolution Options

Option 1: Implement Robust XML Buffer Management (Recommended)

• What: Replace streaming chunk parsing with complete response accumulation
• Where: Modify AIResponseParser.parseChunk() to buffer complete XML blocks before parsing
• Why: Eliminates XML fragment corruption at chunk boundaries
• Implementation:
  • Buffer streaming responses until complete </item> tags are found
  • Only parse complete XML structures
  • Add XML validation before processing
• Pros: Fixes core streaming issue, maintains performance
• Cons: Slightly higher memory usage

Option 2: Add Transaction-Based Translation State

• What: Implement atomic translation operations with rollback capability
• Where: Add state management to TranslateStrings.php translation loop (lines 340-385)
• Why: Ensures partial translations don't corrupt the target files
• Implementation:
  • Save translation state before each chunk
  • Validate complete translation set before committing
  • Rollback and retry on partial failure
• Pros: Prevents corruption from reaching output files
• Cons: More complex implementation, potential performance impact

Option 3: Enhanced Verification with Auto-Retry Logic

• What: Strengthen the verify() method and add intelligent retry logic
• Where: Modify AIProvider.verify() and add retry logic in TranslateStrings.php
• Why: Catches incomplete translations and retries automatically
• Implementation:
  • Require 100% key completion in verification
  • Add exponential backoff retry with different chunk sizes
  • Implement context reduction on retry
• Pros: Addresses user's main complaint about multiple manual runs
• Cons: May increase API costs, doesn't fix root XML parsing issue

Recommendation

Option 1 is the most effective solution as it addresses the core streaming XML parsing issue that causes both incomplete translations and XML corruption. This should be combined with improved error logging to help diagnose future issues.

---