Save money. Fix regressions.

One "rag.answer" produces ~45 "openai.chat.completions.create" calls — almost always a per-retrieved-item LLM call. Batch them into one request, or cap the loop. 45 completions for a single answer is a retrieval/ranking bug, not a model problem.

# One operation made N LLM calls in a single trace.
# Batch the per-item calls into one request:
inputs = [c.text for c in retrieved[:TOP_K]]   # e.g. TOP_K = 8
client.embeddings.create(model="text-embedding-3-small", input=inputs)

# Or collapse a per-chunk loop into a single completion:
context = "\n\n".join(c.text for c in retrieved[:TOP_K])
client.chat.completions.create(messages=[{"role": "user", "content": context}])

Short prompts don't need a frontier model. Add a length check at the dispatcher: if tokens_input < 600, use gpt-4o-mini; otherwise fall back. Quality drop is typically negligible at this length.

# Route short prompts to cheaper model
model = "gpt-4o-mini" if tokens < 600 else "gpt-4o"
client.chat.completions.create(model=model, ...)

At this volume a fine-tuned smaller model typically reaches ≥95% of frontier quality on a constrained task. Sample 5k spans, fine-tune gpt-4o-mini, A/B against current. Training cost recovers in ~5 days at current spend.

# Export training data from Peekr spans
# then fine-tune gpt-4o-mini on your task

Anthropic prompt caching cuts repeated system-prompt cost by ~90% after the first hit. Set cache_control: {"type":"ephemeral"} on the system block — no code path change required on Peekr's side.

# Anthropic — cache the system prompt
client.messages.create(
    system=[{"type": "text", "text": prompt,
             "cache_control": {"type": "ephemeral"}}],
    ...
)

A low-faithfulness answer costs the same to generate as a correct one — then costs you again in retries, support, and lost trust. Tighten grounding (better retrieval, require citations), or block-and-retry: Peekr's HallucinationBlock guard can withhold an answer scoring below your floor before it reaches the user.

# Stop serving (and paying for) unsupported answers
peekr.instrument(
    evaluators=[peekr.eval.Hallucination()],
    guards=[peekr.guard.HallucinationBlock(min_score=0.6)],  # withhold + retry
)

At this volume a fine-tuned smaller model typically reaches ≥95% of frontier quality on a constrained task. Sample 5k spans, fine-tune gpt-4o-mini, A/B against current. Training cost recovers in ~5 days at current spend.

# Export training data from Peekr spans
# then fine-tune gpt-4o-mini on your task

Gate the expensive call on the triage score: if the router scores an input "minimal" or "ignore", short-circuit before the model call (a templated reply, a cached answer, or nothing). You're paying frontier-model rates to think about messages you already decided to skim.

# Gate the expensive call on the triage score
level = router.score_attention(message)        # "full" | "minimal" | "ignore"
if level in ("minimal", "ignore"):
    return cheap_reply(message)                # template / cache — no model call
answer = client.chat.completions.create(...)   # only when it's worth it

A low-faithfulness answer costs the same to generate as a correct one — then costs you again in retries, support, and lost trust. Tighten grounding (better retrieval, require citations), or block-and-retry: Peekr's HallucinationBlock guard can withhold an answer scoring below your floor before it reaches the user.

# Stop serving (and paying for) unsupported answers
peekr.instrument(
    evaluators=[peekr.eval.Hallucination()],
    guards=[peekr.guard.HallucinationBlock(min_score=0.6)],  # withhold + retry
)

Each call waited for the previous one to finish. Running them concurrently (ThreadPoolExecutor in Python, Promise.all in JS) reduces wall-clock time from 42.3s to ~1.2s — a 35.8× speedup with no change to cost or output quality.

# Python — run LLM calls in parallel
from concurrent.futures import ThreadPoolExecutor

def process_all(items):
    with ThreadPoolExecutor(max_workers=8) as pool:
        return list(pool.map(process_one, items))