RAG chunking: split by tokens, not characters

Every embedding model has a token limit. Most RAG chunking code I see enforces it with character counts, on the theory that a token is "about four characters." That's true on average and wrong exactly when it matters: tables, code, identifiers, and non-English text all tokenize far worse than prose.

When a chunk goes over the limit, depending on the provider you get an error, a truncation, or a silently degraded vector. The last one is the worst, because retrieval just gets a little bit worse and nobody knows why.

Text chunking: count what the model counts

The fix is to split on the tokenizer's own output, then batch the pieces into a single embedMany call with the AI SDK:

generate-embeddings.ts

import { embedMany } from 'ai'
import { encode, decode } from 'gpt-tokenizer'
 
const MAX_TOKENS = 8000
 
export async function embedValues(values: string[]) {
  const work: { text: string; index: number }[] = []
 
  values.forEach((value, index) => {
    const tokens = encode(value)
    for (let i = 0; i < tokens.length; i += MAX_TOKENS) {
      work.push({ text: decode(tokens.slice(i, i + MAX_TOKENS)), index }) 
    }
  })
 
  const { embeddings } = await embedMany({
    model: 'openai/text-embedding-3-small',
    values: work.map((w) => w.text),
  })
 
  return work.map((w, i) => ({ index: w.index, embedding: embeddings[i] }))
}

Keep the original index

The line that matters is carrying index through. One input value may become several chunks, and the embeddings come back as a flat list. Without the original index you can't recombine them, so you can't tell which record a retrieved chunk belongs to.

With it, you can store one row per chunk that points back to its source, or average a record's chunk vectors if you need exactly one embedding per record.

Why this is worth a post

It's a small change. But it's representative of most real RAG work: the model is rarely the problem. The plumbing around it, where you split, what you keep, how you recombine, decides whether retrieval is good.