17.11 · When Properties Collide — Chẩn đoán khi 2 thuộc tính giao thoa

Mở đầu: Một trải nghiệm quen thuộc

Tìm 5 fintech startups Vietnam lớn nhất, founded 2020-2024. Cho tôi: tên, founder, tổng funding raised, HQ location, và 1 câu về product.

Claude trả về table đẹp:

Bạn paste vào deck, gửi sếp. Sếp hỏi: "Tiki có làm fintech à? Thủa nào vậy?"

Bạn Google. Tiki không có sản phẩm lending tên 'Finova'. Thông tin bịa. Nhưng 4 công ty kia đúng. Mix-up giữa thật và bịa trong cùng output.

Cái gì đã hỏng?

Câu trả lời 1 thuộc tính: "Hallucination" (Next Token Prediction).

Câu trả lời 2 thuộc tính — chính xác hơn: Next Token Prediction × Knowledge.

• Knowledge gap: Vietnamese fintech niche + recent → thin coverage.
• NTP mechanism: Model fill gap bằng cách sinh text-shape-like-a-fact trong cùng structure với facts thật.

Khi bạn đặt tên 2 thuộc tính colliding, fix trở nên obvious:

• Fix NTP-only: "verify specifics" → hữu ích nhưng manual.
• Fix Knowledge-only: "use web search" → đã dùng nhưng vẫn bịa.
• Fix Collision: use source grounding strict — force model chỉ cite từ retrieved docs, không từ training. Nếu không tìm được → say "not found", không fabricate.

Một property = generic fix. Hai properties = targeted fix.

Bài này dạy bạn diagnostic skill này.

Four properties — nhắc lại nhanh

Trước khi diagnose collision, nhắc 4 properties (nếu cần ôn, quay lại Bài 17.3, 17.6, 17.8, 17.10):

🔮 Next Token Prediction — "Generates what sounds right"
    Predicts text-shape, không phải lookup facts.

🌐 Knowledge — "Knows what it was trained on"
    Broad but uneven, frozen at cutoff.

🧠 Working Memory — "Attends to what's nearby"
    Fixed context window, cliff failure.

🎚️ Steerability — "Follows the loudest instruction"
    Pattern-matches instructions, not intent.

5 collision patterns phổ biến

Collision 1: Next Token Prediction × Knowledge = Hallucinated specifics

Symptom: Hỏi về niche topic → get paper title, author names, journal, DOI — none real.

Cơ chế:

• NTP: Model đang generate text fitting "what a plausible citation looks like"
• Knowledge: Có gap (niche or post-cutoff)
• Model không biết phân biệt what it knows vs what it's inventing

Ví dụ mạch lạc:

User: "Cite 3 papers on X (niche topic)"
Model: 
  - [Plausible-sounding citation 1]     ← maybe real
  - [Plausible-sounding citation 2]     ← fabricated
  - [Plausible-sounding citation 3]     ← fabricated

Fix (targeted):

• Source grounding: force retrieve from real document corpus → model can only cite retrieved
• Explicit "tag fabricated": "For each citation, tag HIGH confidence (verified in your training) or LOW (inferring)"
• Use web search + verify: Research mode Claude, Perplexity
• Don't: prompt "don't fabricate" — model doesn't know it's fabricating

Collision 2: Working Memory × Steerability = Long-conversation drift

Symptom: Set careful constraints at start; 20 messages later, half are being ignored.

Cơ chế:

• Working Memory: Early context fading (lost in middle as chat grows)
• Steerability: Follows whatever instructions most salient right now
• Early constraints → distant → less salient → not applied

Ví dụ:

Turn 1: "Always respond in markdown. Never use first person. 
         Max 200 words."

Turns 2-30: Normal chat, various topics.

Turn 31: Response in prose, uses "I think", 400 words.
Constraints still in context, but attention weight diluted.

Fix (targeted):

• Re-supply critical context: restate constraints near current message
• Start fresh với essentials up front (better than long drifted chat)
• Use system prompt for standing constraints (doesn't dilute)
• Use Projects/Custom Instructions — persists across chats
• Don't: assume "early instructions still apply"

Collision 3: Next Token Prediction × Steerability = Letter-over-spirit at scale

Symptom: Instructions honored literally, intent consistently missed.

Cơ chế:

• NTP: Pattern-match on words
• Steerability: Follow instruction without understanding goal
• Model picks easiest-to-match interpretation

Ví dụ:

User: "Make these 20 emails more professional."
Model: Adds formal language to all 20.

But: 5 of the 20 had the opposite problem — too formal, 
    losing warmth with long-term customers.
    
Model applied one interpretation uniformly.

Fix (targeted):

• State intent per cluster: "These 15 emails: more formal. These 5: warmer."
• IPO framework (Bài 17.10)
• Chain-of-thought: ask model reason about each case
• Don't: rely on single instruction for heterogeneous task

Collision 4: Knowledge × Steerability = Confidently wrong on niche instructions

Symptom: Give domain-specific instruction → AI follows but produces wrong output because lacks domain knowledge.

Cơ chế:

• Knowledge: Gap in specialized domain
• Steerability: Follows instruction confidently anyway
• Result: Confident wrong output (no signal of knowledge gap)

Ví dụ:

User: "Write a unit test for this Python function using our 
       internal test framework 'FrameX'."
       
Model: [writes test with pytest-like syntax that LOOKS like 
       FrameX but isn't actually FrameX]
       
Model didn't flag: "I'm not familiar with FrameX."

Fix (targeted):

• Provide domain context: paste FrameX docs into prompt
• Ask to verify knowledge: "Do you know FrameX? If not, flag."
• Show example first: give 1-2 examples of correct FrameX tests
• Don't: assume model "probably knows"

Collision 5: All 4 — Long multi-step task with specific domain

Symptom

Task is long (Working Memory), domain-specific (Knowledge), multi-step (Steerability drift), requires specifics (NTP hallucination). Everything fails.

Ví dụ

"Review 50-page medical contract between pharma company and CRO, flag compliance issues under 21 CFR Part 11."

• Working Memory: 50 pages stretching context
• Knowledge: 21 CFR Part 11 niche + regulatory
• Steerability: Multi-step review drifts
• NTP: Specifics (sections, regulatory references) hallucination risk

Fix (layered):

• Chunking (WM fix)
• RAG with 21 CFR docs (Knowledge fix)
• Checkpoints per chunk (Steerability fix)
• "Cite exact section from contract" (NTP fix)
• Human legal review of output (all fix)

Diagnostic framework: "Which 2 collided?"

Trước khi reach for prompt fix, ask:

┌──────────────────────────────────────────────────────────┐
│                                                          │
│  DIAGNOSTIC CHECKLIST:                                   │
│                                                          │
│  1. What property question was most relevant?            │
│     - Where do answers come from? (NTP)                  │
│     - What does it know? (Knowledge)                     │
│     - What's it attending to? (WM)                       │
│     - How much am I in control? (Steerability)           │
│                                                          │
│  2. What was the FAILURE MODE?                           │
│     - Fabricated specifics? → NTP                        │
│     - Stale / wrong facts? → Knowledge                   │
│     - Ignored earlier info? → Working Memory             │
│     - Literal interpretation miss intent? → Steerability │
│                                                          │
│  3. PAIR them up — most failures involve 2:              │
│     - Fabricated + niche domain → NTP × Knowledge        │
│     - Drifted + long session → WM × Steerability         │
│     - Wrong output + specific instruction → K × S        │
│     - Literal + varied cases → NTP × Steerability        │
│                                                          │
│  4. Apply TARGETED fix:                                  │
│     - NTP: verify specifics, use grounding               │
│     - Knowledge: add context, use RAG/search             │
│     - WM: re-supply context, chunk, start fresh          │
│     - Steerability: restate goal, checkpoints, examples  │
│                                                          │
└──────────────────────────────────────────────────────────┘

Diagnostic table: Common symptoms → properties → fixes

Ví dụ theo ngành

Sales Manager — "AI biến một deal thành 2 deals"

Situation

Sales manager dùng Claude để summarize 20 sales call transcripts cho team review.

Output

Claude cho 15 deal summaries — but 2 của chúng hợp nhất thành 1 deal mới bịa ra, và split 1 real deal thành 2 riêng biệt.

Diagnostic:

• Working Memory: 20 transcripts ~80K tokens, pushed into context length limits
• NTP: Generating summary with normalized structure — model "filled pattern" instead of preserving unique deals

Collision: WM × NTP

Fix:

• Chunk: 5 transcripts per pass
• Per pass: extract structured output {deal_id, summary, status}
• Aggregate at end
• Verify count (20 in, 20 out)

Kết quả: 0 lỗi merge/split. Process 2x faster.

Research Analyst — "Lit review bịa 3 paper"

Situation

Academic researcher nhờ Claude review 40 papers về emerging field.

Output

Excellent synthesis — but 3/40 paper names cited don't exist.

Diagnostic:

• Knowledge: emerging field → thin coverage
• NTP: generating citations in academic format → fabrication pattern

Collision: Knowledge × NTP (classic)

Fix:

• Upload 40 PDFs into Claude Projects
• Prompt: "Synthesize based ONLY on uploaded papers. Cite by [Paper #]. If unsure, flag [NEEDS VERIFY]"
• After draft: verify every [Paper #] references uploaded doc
• No citations from "training memory"

Kết quả: 100% citation accuracy.

Legal Counsel — "50-page contract review — AI missed 3 critical clauses"

Situation

Senior counsel review 50-page M&A contract via Claude.

Output

Found 8 issues — but missed 3 critical indemnification clauses buried in sections 4-6 (middle).

Diagnostic:

• Working Memory: "lost in the middle" — sections 4-6 deep in attention dead zone
• Steerability: instruction "find issues" was generic, drifted during long review

Collision: WM × Steerability

Fix:

• Chunk 50 pages into 5 × 10-page sections
• Per chunk: specific instruction — "Find all indemnification, limitation of liability, termination clauses. Flag any deviation from market standard."
• Aggregate at end with dedupe

Kết quả: Caught all 11 issues (including 3 missed earlier). Deal saved $1.5M risk.

Finance Analyst — "Wrong IRR in memo"

Situation

Analyst asks Claude: "Calculate IRR for project with these cashflows: -100, 30, 35, 45, 50. Write investment memo with IRR + NPV at 10% discount."

Output

"IRR = 18.4%, NPV = $35.72M" (both approximate, off from actual).

Diagnostic:

• NTP: arithmetic weakness (approximating by patterns)
• Steerability: instruction "calculate" was literal but model lacked mechanism for precision

Collision: NTP × Steerability (brittle arithmetic)

Fix:

• Code execution: model write Python, execute, use exact number
• Prompt: "Compute using Python. Present exact number. Don't approximate."

IRR = 18.73%
NPV = $36.23M

Kết quả: Precision 100%. Memo credible.

Customer Success Manager — "Chat got worse over 30 messages"

Situation

CSM starts chat với Claude to draft renewal proposals for 10 clients. Long session, refining each.

Output

First 5 proposals — excellent, on-brand. Last 5 — drift: inconsistent tone, mixing brand styles, missing structure element from earlier.

Diagnostic:

• Working Memory: early style guide + examples (Turn 1-3) now deep in history
• Steerability: latest instruction salience trumps earlier structure
• Minor: fingerprint verbosity leaking in as attention diffuses

Collision: WM × Steerability

Fix:

• Setup Claude Project với style guide + brand examples as standing docs
• Per proposal: new chat trong Project (context clean, style reference loaded)
• If long session: handoff doc every 5 proposals

Kết quả: Consistency 10/10 proposals.

Clinical Research Coordinator — "AI gave wrong drug protocol"

Situation

CRC asks Claude about study drug dosing protocol for Phase III trial.

Output

Claude confidently states protocol. Later, checking with PI: several details wrong.

Diagnostic:

• Knowledge: Phase III trials are specific; protocol details niche
• Steerability: Claude followed "tell me the protocol" without flagging uncertainty
• Over-caution fingerprint absent when it should've been present (ironically)

Collision: Knowledge × Steerability

Fix:

• RAG: upload study protocol PDF
• Prompt: "Answer ONLY from uploaded protocol. If info not in protocol, say 'Not in uploaded documents.'"
• Never ask protocol questions without upload

Kết quả: 0 errors. CRC can trust outputs.

Anti-patterns

"It's just hallucination" — single-property diagnosis

Tại sao sai

Too generic → fix too generic.

Cách đúng

Name 2 properties. "NTP × Knowledge gap on niche topic" → targeted fix (RAG + grounding).

"Re-prompt until it works"

Tại sao sai

Random prompting. Same failure likely repeat. Waste time.

Cách đúng

Diagnose first. Different failure types need different fixes.

"Model got better, same prompt works now"

Tại sao sai

Maybe. But limitations shifted, not disappeared. Collision patterns still exist.

Cách đúng

Keep diagnostic habit. Test periodically. Boundaries move.

"Single fix = solves all my problems"

Tại sao sai

Different tasks have different property profiles. One fix (e.g., "always use RAG") helps some tasks, hurts others.

Cách đúng

Per-task diagnosis. Right tool for right collision.

"AI will eventually ask me if confused"

Tại sao sai

Model often picks interpretation without asking (Steerability + NTP collision).

Cách đúng

Instruct upfront: "If ambiguous, ask before proceeding."

Mẹo nâng cao

Mẹo 1: "Failure log" với property tagging

Maintain log:

Date | Task | Failure | Properties | Fix applied | Result

2026-03-01 | Lit review | Bịa 3 citations | NTP × Knowledge | 
  RAG with PDFs | 100% accurate

2026-03-05 | Long contract review | Missed middle clauses | 
  WM × Steerability | Chunking | Caught all

2026-03-10 | Email redraft | Letter-over-spirit | NTP × Steerability | 
  IPO framework | Better

Over weeks, patterns emerge. Apply preemptively.

Mẹo 2: Task complexity matrix before starting

For each task, rate (1-5):
  - Specificity required (NTP risk)
  - Domain niche-ness (Knowledge risk)
  - Content length (WM risk)
  - Instruction complexity (Steerability risk)

Total > 12 → plan for collision. Use RAG + chunking + checkpoints.
Total < 8 → low risk, standard prompt.

Mẹo 3: "Stress test" bằng cách introduce property issues

To test new prompt:

Test 1 (WM stress): Add 30 irrelevant turns before.
Test 2 (Knowledge stress): Ask about obscure niche variant.
Test 3 (Steerability stress): Give contradicting constraints.
Test 4 (NTP stress): Ask for 5 very specific facts.

See which breaks first → know your weakest property for this task.

Mẹo 4: "Property-aware review" habit

After every AI output:

Quick 30-second review:
  ✓ Specifics (names, numbers, citations): verified?
  ✓ Knowledge currency: up-to-date?
  ✓ All input context attended: nothing missed?
  ✓ Intent (not just instruction): achieved?

If no → identify collision → fix specifically.

Mẹo 5: Cross-property failure cascades

Some failures cascade:

Initial: Bịa specifics (NTP)
  → Include in longer doc
  → Long doc → WM limit
  → User skim, miss that specifics bịa (NTP amplified)
  → Act on wrong info

Prevent cascades: catch failures early, one layer at a time.

Áp dụng ngay

Bài tập 1: Failure Diagnosis (~25 phút)

Lý do: Most real-world AI failures aren't 1 property acting up. They're 2 properties meeting. Naming which 2 changes the fix entirely.

Step 1: Nhìn lại experience với AI (kể cả quan sát trong khóa học này). Identify 2-3 lần AI output thực sự làm bạn disappointed hoặc surprised.

For each, describe trong 1-2 câu: what bạn asked, what bạn got, what was disappointing/surprising.

Failure 1: ________________________________
Failure 2: ________________________________
Failure 3: ________________________________

Step 2: Walk through each event with the AI:

Based on the 4 properties (Next Token Prediction, Knowledge, Working Memory, Steerability), which ones were likely at play here, and why?

Step 3: Evaluate AI's diagnosis against what bạn giờ biết. Do you agree? If not, push back.

Remember sycophancy fingerprint (Bài 17.2): AI có thể agree với framing of bạn quá readily. If you think nó wrong, say so.

Step 4: For each diagnosis:

Given that diagnosis, what's the most targeted fix?

If possible, test the adjustment right now on a similar task.

Step 5: Nhìn lại task list từ Bài 17.0 với tất cả annotations:

• Property tags (Bài 17.1)
• Verification scores (Bài 17.3)
• Knowledge flags (Bài 17.5)
• Context needs (Bài 17.7)
• Goal statements (Bài 17.9)

For tasks gave you trouble most, name which 2 properties were colliding. Write diagnosis next to each.

Bài tập 2: Preemptive collision mapping (optional)

For your top 5 recurring AI tasks, complete:

Apply setups. Track whether collision avoided.

Suy ngẫm bài học

• Naming the property pair có change fix bạn'd reach for không? Before this course, bạn có chọn different (less effective) fix không?
• Which property pairing bạn nghĩ you'll gặp most in day-to-day work?
• Có failure nào từ trước mà giờ bạn'd gọi tên differently?

Tóm tắt bài học

Tài liệu tham khảo

• Bài 17.3, 17.6, 17.8, 17.10 — 4 properties in detail
• Anthropic — AI Fluency course — Complementary 4D Framework
• Bài 17.12 — Putting it all together