6.38 · Fine-grained tool calling — Stream tool inputs

Mở đầu: Tool streaming cần nhanh hơn

Streaming tool = Claude gửi partial JSON args từng chunk, bạn hiển thị progress cho user.

Nhưng mặc định Anthropic validate JSON trước khi send → có delay. Fine-grained mode skip validation → stream faster.

Basic tool streaming

Với stream=True khi có tools:

with client.messages.stream(
    model=model,
    max_tokens=1000,
    messages=messages,
    tools=[save_article_schema],
) as stream:
    for event in stream:
        if event.type == "content_block_start":
            if event.content_block.type == "tool_use":
                print(f"\n→ Calling tool: {event.content_block.name}")
        
        elif event.type == "input_json":
            # Partial JSON chunk
            print(event.partial_json, end="", flush=True)
        
        elif event.type == "content_block_stop":
            pass

InputJsonEvent properties:

• partial_json — chunk JSON mới
• snapshot — cumulative JSON so far

Default behavior: validated streaming

Anthropic buffer JSON chunks, validate complete top-level key-value, rồi send.

Ví dụ

Tool input schema expect:

{
  "abstract": "...",
  "meta": {
    "word_count": 847,
    "review": "..."
  }
}

Flow:

1. Claude generate "abstract": "This paper"... (incomplete)
2. Anthropic buffer
3. Claude hoàn thành "abstract" value
4. Anthropic validate against schema → OK
5. Send all abstract chunks together
6. Claude generate "meta": {...} → validate → send

Observation

User thấy: pause → burst → pause → burst. Không smooth as text streaming.

Fine-grained tool calling

Flag: anthropic-beta: fine-grained-tool-streaming-2025-05-14 (check docs for latest version).

Hoặc parameter fine_grained=True (tùy SDK version).

Effect

• Disable validation server-side
• Chunks sent immediately as Claude generates
• True streaming behavior

Trade-off

JSON có thể invalid khi nhận — bạn phải handle.

When to use fine-grained?

Use when

• Real-time progress UI (user thấy "typing..." effect)
• Long tool args (500+ words in 1 field)
• Start processing partial data before completion

Don't use when

• Short tool args (validation overhead minimal)
• JSON strict required downstream
• Không muốn handle invalid JSON

Code example

import json

with client.messages.stream(
    model=model,
    max_tokens=2000,
    messages=messages,
    tools=[save_article_schema],
    extra_headers={
        "anthropic-beta": "fine-grained-tool-streaming-2025-05-14"
    }
) as stream:
    current_args = {}
    
    for event in stream:
        if event.type == "input_json":
            # partial_json có thể invalid (e.g., "word_count": undefined)
            try:
                parsed = json.loads(event.snapshot)
                current_args = parsed
                # Use partial data
                print(f"\rProgress: {len(str(current_args))} chars", end="")
            except json.JSONDecodeError:
                # Expected với fine-grained
                pass

UI pattern: Progressive tool call display

async def stream_with_progress(messages, tools):
    tool_name = None
    accumulated_args = ""
    
    with client.messages.stream(...) as stream:
        for event in stream:
            if event.type == "content_block_start" and event.content_block.type == "tool_use":
                tool_name = event.content_block.name
                await ui.send_event({"type": "tool_start", "name": tool_name})
            
            elif event.type == "input_json":
                accumulated_args += event.partial_json
                # Try parse và send progress
                try:
                    args = json.loads(accumulated_args)
                    await ui.send_event({
                        "type": "tool_progress",
                        "args_so_far": args
                    })
                except:
                    pass
            
            elif event.type == "content_block_stop":
                await ui.send_event({"type": "tool_complete", "name": tool_name})

Frontend show:

→ Calling: save_article
  Abstract: "This paper presents..."      [typing...]
  Word count: 847                          [complete]
  Review: "Novel approach to..."           [typing...]

Feel responsive, smooth.

Validation vẫn happen cuối cùng

Fine-grained chỉ skip validation trong streaming. Khi Claude kết thúc tool_use block, final input phải valid JSON matching schema.

Nếu invalid → error ở client:

tool_call = response.content[0]
args = tool_call.input  # ← đây phải là valid dict

Combined with text streaming

Response có thể có cả text + tool:

with client.messages.stream(...) as stream:
    for event in stream:
        if event.type == "content_block_delta":
            if event.delta.type == "text_delta":
                print(event.delta.text, end="", flush=True)
            elif event.delta.type == "input_json_delta":
                # Fine-grained tool chunk
                ...

Modern UX: Claude "talk" (text) while "preparing tool call" (tool args streaming).

Anti-patterns

Dùng fine-grained không cần

Short tool input → no benefit, extra complexity.

Fix: Default mode OK cho 95% case.

Assume JSON valid mid-stream

parsed = json.loads(event.snapshot)  # crash với fine-grained

Fix: Wrap try-except.

Block UI khi tool streaming

Render tool args chặn main thread.

Fix: Async rendering.

Áp dụng ngay

Bài tập 1: Basic tool streaming (20 phút)

Build tool generate_article(topic, length) — simulate generate dài.

Test default mode — observe delay patterns.

Bài tập 2: Fine-grained (30 phút)

Enable fine-grained. So sánh TTFT (time to first token):

• Default: ~2s (wait validation)
• Fine-grained: ~0.3s

Handle invalid JSON gracefully trong UI.

Tóm tắt bài học