Cost intelligence

One "rag.answer" produces ~45 "openai.chat.completions.create" calls — almost always a per-retrieved-item LLM call. Batch them into one request, or cap the loop. 45 completions for a single answer is a retrieval/ranking bug, not a model problem.

# One operation made N LLM calls in a single trace.
# Batch the per-item calls into one request:
inputs = [c.text for c in retrieved[:TOP_K]]   # e.g. TOP_K = 8
client.embeddings.create(model="text-embedding-3-small", input=inputs)

# Or collapse a per-chunk loop into a single completion:
context = "\n\n".join(c.text for c in retrieved[:TOP_K])
client.chat.completions.create(messages=[{"role": "user", "content": context}])

Short prompts don't need a frontier model. Add a length check at the dispatcher: if tokens_input < 600, use gpt-4o-mini; otherwise fall back. Quality drop is typically negligible at this length.

# Route short prompts to cheaper model
model = "gpt-4o-mini" if tokens < 600 else "gpt-4o"
client.chat.completions.create(model=model, ...)

At this volume a fine-tuned smaller model typically reaches ≥95% of frontier quality on a constrained task. Sample 5k spans, fine-tune gpt-4o-mini, A/B against current. Training cost recovers in ~5 days at current spend.

# Export training data from Peekr spans
# then fine-tune gpt-4o-mini on your task