6.51 · Nhập môn RAG — Retrieval Augmented Generation

Mở đầu: 800 trang tài liệu, câu hỏi đơn giản

Risk factors của công ty là gì?

Options:

Option 1: Stuff — Dump toàn bộ

prompt = f"""Answer based on document.

<question>{user_q}</question>
<document>
{800_pages_of_text}
</document>"""

Vấn đề:

• 800 trang ≈ 400K tokens → có thể vượt context window (Sonnet 1M OK, Opus 200K fail)
• Cost: 400K × $3/M = $1.20 mỗi query
• Latency: 30-60s processing
• Quality: Claude "loãng" với context dài — miss info

Option 2: RAG — Chunk + Retrieve

1. Chunk doc thành 2000 pieces × 500 tokens mỗi
2. Khi user hỏi, tìm top 5 chunks liên quan
3. Chỉ gửi 5 chunks (2500 tokens) vào prompt

Benefits:

• Cost: 2500 × $3/M = $0.008 mỗi query (150x cheaper)
• Latency: 2-3s
• Quality: focused context → Claude trả lời chính xác
• Scale: work với 1000+ documents

Trade-off: Setup RAG pipeline phức tạp hơn.

Công thức RAG

┌──────────────────────────────────────────────────┐
│                                                  │
│  PREPROCESSING (một lần):                        │
│     1. Chunk docs thành pieces                   │
│     2. Generate embeddings cho mỗi chunk         │
│     3. Store vào vector database                 │
│                                                  │
│  QUERY TIME (mỗi request):                       │
│     4. Embed user question                       │
│     5. Search similar chunks (top K)             │
│     6. Include top K chunks in prompt            │
│     7. Send to Claude → response                 │
│                                                  │
└──────────────────────────────────────────────────┘

3 bước preprocessing (one-time), 4 bước runtime (mỗi query).

Khi nào dùng RAG?

Cần RAG khi

• Document > 100K tokens
• Multi-doc corpus (search across all)
• Cost-sensitive (high query volume)
• User questions unpredictable (can't pre-select relevant sections)
• Long-term knowledge base (docs update, re-index)

Không cần RAG khi

• Doc < 20K tokens → stuff vào prompt OK
• Static doc, few queries → cost stuff is fine
• Need holistic view (summarize entire doc) → stuff
• Q&A về structure global → RAG miss picture

Consider hybrid

• Doc 100K-1M tokens: dùng Sonnet 1M context + cache. Có thể không cần RAG.

Benefits vs Challenges

Ví dụ: Corporate wiki

Setup

Company wiki: 5000 articles, 2M tokens total.

Without RAG

Impossible — exceed all context windows, costs 100x.

With RAG

• Preprocessing: chunk 5000 articles → 20K chunks. Embed. Store.
• Query: "Policy về WFH?" → retrieve top 5 relevant chunks → send to Claude.
• Cost per query: $0.01
• Response time: 2-3s

100 users, 10 queries each/day = $10/day. Affordable.

Ví dụ: Customer support KB

Scenario

500 product docs. Customer asks: "Why is my device not charging?"

RAG flow

1. Embed question
2. Retrieve: "Troubleshooting charging issues" article (top match)
3. Claude receives article + question → response
4. Claude cite article source

Result: Accurate answer, source-backed, scale qua 500 docs.

Technical decisions trong RAG

You'll decide:

1. Chunk size (500-2000 tokens typical)
2. Chunk overlap (10-20% for context preservation)
3. Chunking method (size-based, sentence, structural, semantic)
4. Embedding model (VoyageAI, OpenAI, Cohere)
5. Vector DB (Pinecone, Chroma, Qdrant, Weaviate)
6. Top K (5-20 chunks retrieve)
7. Re-rank (use Claude or dedicated reranker to refine top K)
8. Hybrid search (semantic + keyword)

Mỗi decision có trade-off. Module 7 cover từng piece.

Anti-patterns preview

RAG cho mọi task

Simple Q&A 1 doc 10 trang → stuff vào prompt, không cần RAG.

Chunk không có overlap

Info cắt giữa câu → chunk missing context.

Top K=1

Chỉ 1 chunk → thiếu context.

No eval on retrieval

Không biết top K có chứa relevant info không.

Full anti-patterns ở mỗi bài tiếp theo.

Áp dụng ngay

Bài tập 1: Assess RAG need (10 phút)

Cho app của bạn:

• Total corpus size: ___
• Queries/day: ___
• Context window model: ___
• Cost per query (stuffed): ___ vs (RAG): ___

Decision: RAG yes/no? Why?

Bài tập 2: Chunk sketch (10 phút)

Lấy 1 sample doc. Plan chunk strategy:

• Method: size/sentence/structure?
• Chunk size: ___
• Overlap: ___

Tóm tắt bài học