fix: handle unclosed tool calls and GLM arg parsing edge cases (#1344)
## Motivation Tool-call requests can hang indefinitely when `max_tokens` truncates generation mid-tool-call. ## Reproduction 1. Send a chat completion with `tools` and a low `max_tokens` (e.g. 65) to Qwen3-0.6B 2. Model generates `<think>...</think>` then starts `<tool_call>` but `max_tokens` cuts it off before `</tool_call>` 3. **Before this fix:** `parse_tool_calls` buffers tokens after `<tool_call>`, generator exhausts, buffered tokens (including `finish_reason`) are silently dropped → stream hangs forever 4. **After this fix:** buffered tokens are flushed as regular text with `finish_reason` propagated → response returns normally with `finish_reason: "length"` Confirmed with fresh local testing: 4 unclosed tool call flushes triggered in a single session. Also confirmed via production logs from Jan 29 (2 occurrences). ## Changes 1. **`parse_tool_calls` unclosed tool call flush** — when the generator exhausts inside an open `<tool_call>` block, flush buffered tokens as regular text and propagate `finish_reason` 2. **GLM regex fix** — match literal `\n` (not escaped `\\n`) between arg tags; handle missing `</arg_value>` via lookahead 3. **7 new unit tests** for `parse_tool_calls` covering unclosed, closed, passthrough, and failed-parse scenarios ## Why It Works - `parse_tool_calls` now has a post-loop check: if `in_tool_call` is still true, it yields the buffered text with the tracked `finish_reason` instead of silently dropping it - The GLM regex now matches real-world output where newlines appear between tags and `</arg_value>` may be absent ## Test Plan ### Manual Testing - Qwen3-0.6B-4bit with `tools` + various `max_tokens` values (61-75) - Confirmed responses return with `finish_reason: "length"` instead of hanging - Log output shows `"generator exhausted inside unclosed tool call, flushing buffered text"` ### Automated Testing - 7 new tests in `test_parse_tool_calls.py` - Full test suite passes (`uv run pytest`) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net> Co-authored-by: Evan <evanev7@gmail.com> Co-authored-by: Jake Hillion <jake@hillion.co.uk> Co-authored-by: rltakashige <rl.takashige@gmail.com>
This commit is contained in:
@@ -757,6 +757,16 @@ def parse_tool_calls(
|
||||
|
||||
if in_tool_call:
|
||||
tool_call_text_parts.append(response.text)
|
||||
if response.finish_reason is not None:
|
||||
logger.info(
|
||||
"toll call parsing interrupted, yield partial tool call as text"
|
||||
)
|
||||
yield GenerationResponse(
|
||||
text=tool_call_start + "".join(tool_call_text_parts),
|
||||
token=0,
|
||||
finish_reason=response.finish_reason,
|
||||
usage=None,
|
||||
)
|
||||
continue
|
||||
# fallthrough
|
||||
yield response
|
||||
@@ -829,7 +839,7 @@ def patch_glm_tokenizer(tokenizer: TokenizerWrapper):
|
||||
|
||||
_func_name_regex = re.compile(r"^(.*?)<arg_key>", re.DOTALL)
|
||||
_func_arg_regex = re.compile(
|
||||
r"<arg_key>(.*?)</arg_key>(?:\\n|\s)*<arg_value>(.*?)</arg_value>",
|
||||
r"<arg_key>(.*?)</arg_key>(?:\n|\s)*<arg_value>(.*?)(?:</arg_value>|(?=<arg_key>)|$)",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user