6.53 · Text embeddings — Biến text thành số

Mở đầu: "Car" và "automobile" same meaning, different words

Embeddings = foundation của semantic search. Chúng biến text thành vector (list of numbers) mà meaning được encoded trong vị trí không gian.

Embedding là gì?

Text → embedding model → list of ~1000-4000 numbers.

"I love coding"  →  [0.23, -0.45, 0.12, ..., 0.67]  (1024 numbers)
"I enjoy programming"  →  [0.21, -0.44, 0.13, ..., 0.66]  (similar!)
"I hate vegetables"  →  [-0.77, 0.33, -0.12, ..., 0.01]  (very different)

Mỗi số represent "một đặc trưng" của text — không interpretable trực tiếp, nhưng tốc độ compute được.

Similar meaning = similar vectors.

VoyageAI — Anthropic recommended

Anthropic chưa cung cấp embedding API. Recommendation: VoyageAI.

Setup

pip install voyageai

VOYAGE_API_KEY="your_key"

Free tier available: sign up voyageai.com.

Code

from dotenv import load_dotenv
import voyageai

load_dotenv()
client = voyageai.Client()


def generate_embedding(text: str, input_type: str = "query") -> list[float]:
    """Generate embedding for text."""
    result = client.embed(
        [text],
        model="voyage-3-large",  # recommended general-purpose
        input_type=input_type  # "query" or "document"
    )
    return result.embeddings[0]


vec = generate_embedding("I love coding")
print(f"Embedding dimension: {len(vec)}")  # 1024
print(f"First 5: {vec[:5]}")

Query vs Document

VoyageAI (and most models) recommend specifying input_type:

• "document" — for chunks stored in DB
• "query" — for user questions

Model apply slightly different encoding → better search quality.

chunk_embeddings = [generate_embedding(c, input_type="document") for c in chunks]

query_embedding = generate_embedding(user_query, input_type="query")

Cosine similarity

Measure similarity giữa 2 embeddings:

import numpy as np

def cosine_similarity(vec1, vec2):
    a = np.array(vec1)
    b = np.array(vec2)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Result:

• 1.0: identical meaning
• 0.9-1.0: very similar
• 0.5-0.8: somewhat related
• 0-0.4: unrelated
• <0: opposite meaning (rare)

Example

vec_car = generate_embedding("I own a car", input_type="query")
vec_auto = generate_embedding("She drives an automobile", input_type="query")
vec_dog = generate_embedding("My dog is friendly", input_type="query")

print(cosine_similarity(vec_car, vec_auto))  # ~0.85 (similar)
print(cosine_similarity(vec_car, vec_dog))   # ~0.15 (unrelated)

Normalization

VoyageAI (and most) auto-normalize vectors to magnitude 1. This means cosine similarity = dot product:

def cosine_similarity_normalized(vec1, vec2):
    return np.dot(vec1, vec2)  # shorter, same result

Faster computation.

Full search example

chunks = [
    "Medical research on XDR-47 bug virus.",
    "Software engineering fixed 100 bugs this quarter.",
    "Marketing team launched new campaign.",
]

chunk_embeddings = [
    generate_embedding(c, input_type="document") for c in chunks
]

query = "How many software bugs were fixed?"
query_emb = generate_embedding(query, input_type="query")

similarities = [
    cosine_similarity(query_emb, ce) for ce in chunk_embeddings
]

best_idx = np.argmax(similarities)
print(f"Best match: {chunks[best_idx]}")
print(f"Score: {similarities[best_idx]:.3f}")

top_k = sorted(enumerate(similarities), key=lambda x: -x[1])[:3]
for idx, score in top_k:
    print(f"{score:.3f} - {chunks[idx]}")

Claude would then answer based on top chunk.

Models & Dimensions

VoyageAI options:

Domain-specific models better for specialized corpora.

Cost

VoyageAI pricing (check site):

• $0.02 - $0.15 per 1M tokens depending model

For 10K chunks × 500 tokens each = 5M tokens. One-time cost: ~$0.50.

Queries ongoing: $0.05 per 1M queries. Very affordable.

Vector database

Store embeddings efficiently:

Simple in-memory

class SimpleVectorStore:
    def __init__(self):
        self.docs = []
        self.embeddings = []
    
    def add(self, content, embedding, metadata=None):
        self.docs.append({"content": content, "metadata": metadata or {}})
        self.embeddings.append(embedding)
    
    def search(self, query_embedding, k=5):
        sims = [cosine_similarity(query_embedding, e) for e in self.embeddings]
        top_k = sorted(enumerate(sims), key=lambda x: -x[1])[:k]
        return [(self.docs[i], sims[i]) for i, _ in top_k]

OK cho < 10K chunks in-memory. Bigger → dedicated vector DB.

Production options

• Pinecone — managed, fast
• Chroma — open source, simple
• Qdrant — open source, scalable
• Weaviate — features rich
• pgvector — Postgres extension (nếu đã có postgres)

Pick based on scale, budget, infrastructure.

Anti-patterns

Mix query và document embeddings

for c in chunks:
    vec = generate_embedding(c, input_type="query")  # should be "document"

Fix: Consistent input_type.

Different models mix

Embed documents với model A, query với model B → incompatible vectors.

Fix: Same model throughout pipeline.

Không normalize before storing

If manually compute, forget normalize → wrong similarity.

Fix: Use library-managed normalization.

Re-embed mỗi query

Re-run preprocessing mỗi query = expensive + slow.

Fix: Preprocessing one-time, store in DB.

Áp dụng ngay

Bài tập 1: Sign up VoyageAI + test (15 phút)

1. Register voyageai.com
2. Get API key, add to .env
3. Run sample code — verify embedding 1024 dim

Bài tập 2: Build simple search (30 phút)

Chunk sample doc (bài 6.52). Embed all. Query "specific thing". Retrieve top 3. Verify relevance.

Tóm tắt bài học