Prompts are data: prompt management and versioning in the database

In the spring of 2023, right after the ChatGPT API opened up, we built an AI tool that helped professionals draft and score social posts. It was one of our first production LLM products, and two decisions from that project, prompt management in the database and per-user token metering, have shown up in nearly everything we've built since.

Prompt management and versioning belong in the database

The first version had prompts in code, like everyone's did. Every wording tweak meant a pull request, a review, and a deploy. The people with the best sense of what a good prompt looked like weren't the people with commit access.

So prompts became records. Each one was a template keyed by where it was used and what kind of job it did:

enum PromptType {
  scoring = 'scoring',
  suggestion = 'suggestion',
}
 
interface PromptTemplate {
  section: string       // which part of the product uses it
  promptType: PromptType
  template: string      // with {{variables}} filled at runtime
  model: string
}
 
async function getPromptTemplate(section: string, promptType: PromptType) {
  return db.promptTemplates.findOne({ section, promptType }) 
}

This did a few things at once:

• Iteration got faster. Prompt changes shipped without a deploy.
• Scoring and generating got separated. A prompt that judges a post and one that writes one have different jobs, and it helps to be able to tune them independently.
• Prompts got history. With timestamps on every record, "why did output get worse on Tuesday" became a question with an answer.

The tradeoff is that prompts are now code that isn't in version control, so treat edits to them with the same care: log changes, and keep a way to roll back.

Meter tokens per user from day one

The second decision: every API response includes token usage, so store it. Every one. Attached to the user who caused it.

// after every completion
await db.chats.updateOne(
  { memberId },
  { $push: { usage: response.usage } }, // prompt, completion, total tokens
)
 
// later: what has this member cost us?
const [totals] = await db.chats.aggregate([
  { $match: { memberId } },
  { $unwind: '$usage' },
  { $group: { _id: null, totalTokens: { $sum: '$usage.total_tokens' } } },
])

It seemed like bookkeeping at the time. It turned out to be the foundation for pricing, rate limits, abuse detection, and deciding which features were worth their cost. Adding it later means you have no history for any of those decisions.

What's changed since

The specifics have moved on. We'd use the AI SDK and a gateway instead of a provider's SDK directly, structured outputs instead of hoping the model returns valid JSON, and evals instead of eyeballing. But both core ideas held up:

Related: LLM routing: race four models instead of guessing one · RAG chunking: split by tokens, not characters