Benchmark
Four policies. Same gold set. Cost and quality on the same rows.
The task file is versioned at apps/api/app/data/benchmark.json.
Mix
| Band | Count | Examples |
|---|
| Easy L1–L2 | 5 | France, HTTP, 17×24, REST |
| Medium L3 | 5 | Merge lists, TCP vs UDP, coin flips |
| Hard L4–L5 | 5 | 1M QPS limiter, Euclid, Go race, peeked A/B |
Policies
| Policy | What it is |
|---|
| always_frontier | Baseline. Every row on the expensive model |
| difficulty | Naive: complexity < 3 cheap, else expensive. No escalation |
| quality | P(quality|small) plus the guard |
| quality_cache | The product: quality-aware plus prefix cache |
What to read
| Field | Good look |
|---|
| saved_pct | High because easy work left economy |
| avg_quality / worst_quality | Close to frontier, worst case does not collapse |
| quality_delta | Near 0 |
| routing_saved vs cache_saved | Two independent mechanisms |
| escalation_rate | Some hard rows recovered after a cheap miss |