Qwen3 TAG_WITH_TAGGED tool format: p.json() fails on array<object> parameter values; partial tool_call leaked to client poisons multi-turn history

[alesha-pro]

Summary

When a tool has a parameter whose JSON schema is array<object> (e.g. [{"type":"web"}]) and the model (Qwen3 family using the TAG_WITH_TAGGED / <tool_call><function=...><parameter=...> format) emits it as the inner value of a <parameter> tag, the autoparser's PEG p.json() fails mid-stream. Two cascading problems follow:

1. Root parse failure. build_tool_parser_tag_tagged uses p.json() for any parameter that doesn't resolve to a string schema. On array-of-object values the PEG parser aborts, the request ends in a 500 with Failed to parse input at pos N: <tool_call>....
2. Partial state leaked to the client. Before the failure, the partial-parse fallback at common/chat.cpp:2158 returns whatever AST was built so far. For streaming (is_partial=true) this surfaces to the client as an SSE delta containing tool_calls[0].function.arguments = "{" (one character). The client, acting correctly per the OpenAI spec, appends this tool_call to conversation history.
3. History poisoning. Every subsequent request replays that history. func_args_not_string at common/chat.cpp:1842-1860 eagerly parses tool_calls[].function.arguments as JSON and throws a 500 on the bare {, so the conversation is now stuck: every follow-up crashes with Failed to parse tool call arguments as JSON: ... unexpected end of input, even though the client did nothing wrong.

Net effect: a single tool-call attempt with a complex parameter permanently wedges the conversation until the client manually strips the assistant message from history.

Name and Version

version: 8759 (2b2cd57de)
built with GNU 11.4.0 for Linux x86_64

HEAD of master as of 2026-04-11, i.e. PR #21216 is included (verified commit 26229755c is in the binary).

Operating systems

Linux

GGML backends

CUDA

Hardware

4x RTX 3090

Models

Qwen3.5-122B-A10B-Q4_K_M (GGUF). The bug is not specific to this model — it reproduces with any model whose template routes to TAG_WITH_TAGGED tool format.

Steps to reproduce

1. Launch the server with jinja enabled and any tool set containing an array<object> parameter:

   llama-server -m Qwen3.5-122B-A10B-Q4_K_M.gguf -ngl 99 -c 163840 -fa on \
                --host 0.0.0.0 --port 8080 --jinja
   

2. Send a chat completion with a tool definition like:

   {
     "type": "function",
     "function": {
       "name": "firecrawl_search",
       "parameters": {
         "type": "object",
         "properties": {
           "query":   {"type": "string"},
           "limit":   {"type": "integer"},
           "sources": {
             "type": "array",
             "items": {
               "type": "object",
               "properties": {"type": {"type": "string", "enum": ["web","images","news"]}},
               "required": ["type"]
             }
           }
         },
         "required": ["query"]
       }
     }
   }

   and a user prompt that naturally requires calling it with sources set.
3. The model emits (captured via a request-dumping proxy in front of llama-server):

   <tool_call>
   <function=firecrawl_search>
   <parameter=limit>
   10
   </parameter>
   <parameter=query>
   pop culture trends USA March April 2026 movies music TV shows celebrities
   </parameter>
   <parameter=sources>
   [{"type": "web"},
   

4. Server responds with 500 in the SSE stream:

   Failed to parse input at pos 518: <tool_call>
   <function=firecrawl_search>
   ...
   <parameter=sources>
   [{"type": "web"},
   

5. Inspection of the SSE stream of that same request shows a preceding delta event:

   {"choices":[{"delta":{"tool_calls":[{"index":0,"id":"...",
     "type":"function","function":{"name":"firecrawl_search","arguments":"{"}}]}}]}

   i.e. arguments = "{" (one character) was already pushed to the client before the stream fails.
6. Any follow-up chat completion that replays the assistant message as conversation history now fails with:

   Failed to parse tool call arguments as JSON:
   [json.exception.parse_error.101] parse error at line 1, column 2:
   syntax error while parsing object key - unexpected end of input; expected string literal
   

Root cause (pointers)

common/chat-auto-parser-generator.cpp around line 367 in build_tool_parser_tag_tagged:

(schema_info.resolves_to_string(param_schema) ?
    p.tool_arg_string_value(p.schema(until_suffix, ..., param_schema, true)) :
    p.tool_arg_json_value(p.schema(p.json(), ..., param_schema, false)) + p.space())

For non-string parameters the generated rule calls p.json(), which appears to be unable to parse JSON whose value is array<object> inside a <parameter> tag context. Possibly related: nested-object backtracking, or lookahead for the closing </parameter> interfering with the JSON grammar. I have not pinned the exact failure site inside p.json() yet — happy to dig further if useful.

The partial-parse fallback at common/chat.cpp:2158-2178 is what surfaces "{" to the client during streaming, and func_args_not_string at common/chat.cpp:1842-1860 is the secondary crash point that poisons the conversation.

Suggested fixes

1. Parser: Either fix p.json() to parse arbitrary nested JSON inside <parameter> tag context, or (simpler) switch the non-string branch to use until_suffix-based capture + a post-parse json::parse on the raw slice. The latter matches how many real-world clients emit structured parameter values and avoids coupling grammar with schema.
2. Streaming safety: is_partial should not emit a tool_calls[].function.arguments delta unless the accumulated content is at least a syntactically valid JSON prefix that the client can safely append to (or, simplest, flush only when the full tool_call is complete). Emitting "{" and then aborting is strictly worse than emitting nothing.
3. Replay tolerance: func_args_not_string should not throw a 500 when a prior tool_calls[].function.arguments in the input history is not valid JSON. It can either leave the string as-is (letting the template render it verbatim) or wrap it for the template. Throwing forces the client to clean up after the server's own broken output.

Notes

Not a duplicate of #20867 (that one was about MAX_REPETITION_THRESHOLD, fixed by #21216, which is already in this build).

[alesha-pro]

Bisected: regression introduced by PR #21216

Confirmed via direct A/B test on the same HEAD (2b2cd57de), same model, same request body, same host:

Test setup

• Reverse-proxy in front of llama-server logs every request and response to disk.
• A previously captured request body (the one that produced the failure above, stream=true, no custom sampling — server defaults) is replayed 20 times via curl.
• Binary A: build/bin/llama-server from HEAD 2b2cd57de.
• Binary B: same HEAD with git revert --no-commit 26229755c applied (removes only PR common : simplify autoparser tagged parser rules #21216, keeps every other commit). Identical cmake flags, including CMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc.
• Both binaries run with the exact same args: -m Qwen3.5-122B-A10B-Q4_K_M ... -ngl 99 -c 163840 -fa on --host 0.0.0.0 --port 8082 --jinja.
• The server is restarted between A and B (fresh KV state, same model file, same context size). No other changes.

Results

Binary	Runs	Pass	Fail (500 Failed to parse input at pos ...)
HEAD 2b2cd57de (PR #21216 applied)	20	15	5 (25%)
HEAD 2b2cd57de + git revert 26229755c	20	20	0 (0%)

Probability of observing 0/20 failures on the reverted binary under the master binary's 25% failure rate: 0.75^20 ≈ 0.003.

All 5 failures on HEAD have the same shape — parser aborts near the start of the <tool_call> block in the model output. Example tails (as returned in the 500 body):

Failed to parse input at pos 498: <tool_call>\n<function=firecrawl_search>\n<parameter=limit>\n15\n</parameter>\n<parameter=query>\nUS pop culture tre...
Failed to parse input at pos 541: <tool_call>\n<function=firecrawl_search>\n<parameter=limit>\n10\n</parameter>\n<parameter=query>\npop culture trends...
Failed to parse input at pos 791: <tool_call>\n<function=firecrawl_search>\n<parameter=limit>\n10\n</parameter>\n<parameter=query>\nUS pop culture tre...
Failed to parse input at pos 539: <tool_call>\n<function=firecrawl_search>\n<parameter=limit>\n10\n</parameter>\n<parameter=query>\npop culture trends...
Failed to parse input at pos 535: <tool_call>\n<function=firecrawl_search>\n<parameter=limit>\n10\n</parameter>\n<parameter=query>\nUS pop culture tre...

And, importantly, the full model output in the originally captured failing SSE stream (from the first comment) was syntactically well-formed Qwen3 tool-call:

<tool_call>
<function=firecrawl_search>
<parameter=limit>
10
</parameter>
<parameter=query>
pop culture trends USA March April 2026 movies music TV shows celebrities
</parameter>
<parameter=sources>
[{"type": "web"}, {"type": "news"}]
</parameter>
</function>
</tool_call>

i.e. the parser is rejecting a valid model output — the regression is in the parser, not in the model's generation. This is reproducible only probabilistically because the failure appears to depend on sampling path / surrounding reasoning length (note the varying pos values), but the regression signal is clean: 25% → 0% just by reverting 26229755c.

Scope of the change in PR #21216. The PR only touches common/chat-auto-parser-generator.cpp:

• Replaces the manual param_schema.type string-detection walk with common_schema_info::resolves_to_string(param_schema), and
• Changes the upper bound of the optional-arg repetition from optional_parsers.size() to -1 (unbounded), to fix MAX_REPETITION_THRESHOLD (2000) breaks tool-calling grammars for tools with many optional parameters #20867.

For a tool with an array<object> parameter like firecrawl_search.sources, both the old code and the new code should route through the p.tool_arg_json_value(p.schema(p.json(), ..., param_schema, false)) branch, so the change in resolves_to_string is semantically equivalent on this case. That leaves the p.repeat(..., 0, -1) change as the most likely trigger — unbounded repetition of a choice that contains a non-string-branch parameter can change where PEG backtracking decides to give up, and may interact with how p.json() reports its furthest match.

(I haven't yet pinned which of the two changes is load-bearing — happy to A/B them separately if a maintainer wants.)

cc @aldehir (PR #21216 author) — this is not a complaint, the fix for #20867 is definitely needed; just flagging for awareness since the regression shows up specifically on tools with non-string parameters.

[alesha-pro]

Update: revert of PR #21216 is NOT a full fix — the parser bug is broader

Ran the same logging-proxy setup described in the previous comment against the reverted build (2b2cd57de + git revert 26229755c). Expected it to behave cleanly based on the 20/20 replay result above. Instead, on live workload with a different tool, I captured this:

Model output (reconstructed byte-for-byte from the SSE stream my proxy teed to disk):

<tool_call>
<function=firecrawl_scrape>
<parameter=url>
https://thehill.com/opinion/white-house/5784809-trump-era-scandals-shameful-behavior/
</parameter>
<parameter=formats>
["markdown"]
</parameter>
<parameter=onlyMainContent>
True
</parameter>
</function>
</tool_call>

This is a well-formed Qwen3 TAG_WITH_TAGGED tool call. All tags balance, parameter names match the tool schema.

What the server reports:

error: Failed to parse input at pos 170: <tool_call>
<function=firecrawl_scrape>
...

Byte 170 of the input lands inside the literal parameter name formats (specifically between form and ats), immediately after the closing </parameter> of the first (url) parameter. So the parser successfully walked through the entire url block — including URL content that contains / and non-trivial characters — and then failed mid-way through matching the second <parameter=...> literal.

Key observations:

1. This reproduces on the reverted binary (i.e. with both the resolves_to_string change and the p.repeat(..., 0, -1) change rolled back). Reverting PR common : simplify autoparser tagged parser rules #21216 reduces the failure rate dramatically for some inputs but does not eliminate the underlying PEG issue.
2. Trigger is broader than array<object>. Here the only non-string parameter is formats: ["markdown"] — an array<string>. The previous repro was sources: [{"type":"web"}] (array<object>). Both fail in the same general area of build_tool_parser_tag_tagged.
3. Failure location is surprising. The parser reports result.end inside a literal parameter-name string, not inside the JSON-value region. That points at the alternative/repeat logic over optional_parsers, not at p.json() per se. When the parser commits to one branch of the optional-parser choice (e.g. the url branch completes), it seems unable to cleanly re-enter the choice for the next iteration, and the furthest-match position gets stranded part-way into the next literal instead of at the start of it.
4. Observable behavior via SSE. The HTTP status is 200 because headers were flushed before the failure; the error arrives as an error event inside the SSE body, after several tool_calls[0].function.arguments partial deltas have already been emitted. Net effect on the client is still the same as in the original report: a syntactically-incomplete arguments string (e.g. {"url":"https://...","formats":["markdown"],"onlyMainContent":) gets appended to history, and the next turn throws a 500 in func_args_not_string.

Mitigations I'm running locally (neither fixes the parser — they just stop the poisoning cascade):

1. common/chat.cpp — made func_args_not_string lenient: LOG_WRN on json::parse failure and leave args as a string instead of throwing. Breaks the secondary 500 on replay.
2. common/chat.cpp:2158-2178 — drop any tool_calls from the partial-parse fallback path during is_partial (keep content/reasoning only). This prevents the server from ever putting a truncated arguments payload on the wire to a streaming client. Graceful degradation: client sees "no tool call produced" and retries, instead of replaying broken state.

With both patches applied on top of the reverted binary, same HauhauCS-finetuned Qwen3.5-122B / same workload / same model output: 500s stop, the request still doesn't produce a usable tool call for the bad inputs (so the parser bug is still there), but the conversation no longer wedges itself. I think patch 2 in particular is worth landing on master regardless of how the parser gets fixed, because it's strictly safer than emitting { to a client.

Repro setup: aiohttp logging proxy in front of llama-server --jinja, teeing every request and SSE response to disk so I can inspect exact bytes. Happy to share the proxy + raw dumps if useful.

[pwilkin]

Thanks, I'll take a look.