Jev benchmark

Offline tool that compares LLMs on ad-detection accuracy, cost, latency, and JSON compliance using real MinusPod transcripts. Produces a Markdown report committed to this repo.

Layout

benchmarks/llm/
  pyproject.toml             # uv project, imports MinusPod modules at runtime
  benchmark.toml.example     # copy to benchmark.toml and fill in (gitignored)
  .env.example               # copy to .env (gitignored)
  src/benchmark/             # source
  data/
    corpus/                  # committed verified episodes
    candidates/              # gitignored work-in-progress captures
    pricing_snapshots/       # committed pricing history
  results/
    raw/                     # calls.jsonl, episode_results.jsonl, responses/<model>.jsonl shards
    report.md                # current report
    report_assets/           # SVG charts referenced by report.md
    archive/                 # explicit snapshots: results/archive/<date>/

Raw response bodies live in one JSONL shard per model (results/raw/responses/<model>.jsonl, lines of {call_id, body}), so the file count stays flat as the corpus grows. Prompts are not stored at all: they reconstruct deterministically from the corpus, and benchmark show-prompt <call_id> verifies the result against the prompt_hash recorded at call time. benchmark show-response <call_id> prints a single raw response body.

Setup

From benchmarks/llm/:

cp benchmark.toml.example benchmark.toml   # edit to fill in your MinusPod base_url
cp .env.example .env                       # fill in MINUSPOD_PASSWORD and provider keys
uv sync                                    # installs deps into benchmarks/llm/.venv

Then run any command via uv run benchmark <cmd>.

The CLI auto-loads benchmarks/llm/.env on startup (resolved relative to the package, not the CWD, so it works from any directory). Shell-exported variables take precedence over .env values.

Requires MinusPod >= 2.0.28 on the server you point at: the benchmark imports create_windows from ad_detector and fetch_litellm_pricing from pricing_fetcher. Before 2.0.28 those modules eagerly imported webhook_service (which pulls jinja2) and bs4, so a clean uv sync could not start the CLI. 2.0.28 deferred both. The benchmark also calls GET /api/v1/feeds/{slug}/episodes/{id}/original-segments, an endpoint added in 2.0.26. Older episodes return 404 until reprocessed.

Transcript source

segments.json for every corpus episode is pulled byte-exact from the source MinusPod instance’s original-segments endpoint. The transcript itself was generated by faster-whisper inside that instance, not by the benchmark. The Whisper config used:

Setting	Value
Model	large-v3
Backend	local (faster-whisper, CUDA GPU)
Compute type	auto (resolves to float16 on CUDA)
Language	en (forced English, not auto-detect)
VAD gap detection	on (start 3.0s / mid 8.0s / tail 3.0s)

The model.transcribe() call shape (from src/transcriber.py):

WhisperModel(model_size="large-v3", device="cuda", compute_type="auto")
model.transcribe(
    audio,
    language="en",
    initial_prompt=<podcast name + SEED_SPONSORS vocabulary>,
    beam_size=5,
    batch_size=<adaptive: 16/12/8/4 by episode length>,
    word_timestamps=True,
    vad_filter=True,
    vad_parameters={"min_silence_duration_ms": 1000, "speech_pad_ms": 600, "threshold": 0.3},
)

This is the call as it stood when the corpus was transcribed. The initial_prompt seeded 26 sponsor terms (AD_VOCABULARY_TERMS, plus the podcast name) so Whisper produced consistent spellings (Athletic Greens rather than AG1, ExpressVPN rather than express vpn). That biased what showed up in the transcript, and therefore what every benchmarked LLM is scored against. 2.96.0 removed the prompt: on the batched pipeline it cost 7-17% of each episode’s speech, so transcripts from current releases have no prompt, fuller coverage, and no spelling bias. The corpus has not been re-transcribed since. Sponsor names are still matched against src/utils/constants.py SEED_SPONSORS (~250 sponsors, each with canonical name, intentional alias spellings, and category).

A second mapping, SPONSOR_ALIASES, is applied after transcription to normalize Whisper mishearings toward the canonical name (a firm -> Affirm, xerox -> Xero, pure tea -> Purity). About 174 such corrections in production. This affects sponsor-name matching but not the transcript fed to the LLM.

Both lists are rendered in full in the generated results/report.md under “Transcript source”. A small canonical-name sample: Athletic Greens, BetterHelp, Squarespace, Shopify, HelloFresh, NordVPN, ExpressVPN, ZipRecruiter, SimpliSafe, Mint Mobile.

Two practical consequences for anyone reading the F1 numbers:

• F1 numbers are upper-bounded by transcript quality. A smaller Whisper model would produce a different (almost certainly worse) ceiling.
• If a model scores well here, it’s scoring against large-v3 transcripts made with the pre-2.96.0 prompt. Deployments running smaller Whisper models should expect lower F1; 2.96.0 and later transcribe without the prompt, which recovers dropped speech (sponsor reads included) but loses the spelling bias, and that combination has not been benchmarked.

Common workflows

Capture a new episode

benchmark capture --episode-url https://podsrv.example.com/ui/feeds/<slug>/episodes/<id>
# edit data/candidates/ep-<slug>-<id>/truth.txt to verify ad markers
benchmark verify ep-<slug>-<id>

Run the benchmark

benchmark refresh-pricing                  # fetch a fresh pricing snapshot
benchmark run                              # auto-fill all gaps, regenerate report
benchmark run --dry-run                    # preview what would run, no API calls
benchmark run --retry-errors               # also retry calls recorded with error

benchmark run always reads [[models]] from benchmark.toml and data/corpus/ for episodes. To restrict scope, edit the config (set deprecated = true) or move episode directories. There are no --model or --episode filters.

Regenerate the report from existing data

benchmark report

Useful after editing the report template. No LLM calls happen.

Snapshot the report

benchmark archive

Copies results/report.md + assets to results/archive/<YYYY-MM-DD>/.

Audit a stored call

benchmark show-response <call_id>          # raw response body from its per-model shard
benchmark show-prompt <call_id>            # rebuild the user prompt from the corpus + verify prompt_hash
benchmark show-prompt <call_id> --snapshot prompts/05-22-206.txt   # verify against a frozen system prompt

show-prompt exits 3 with a MISMATCH warning when the recomputed hash differs from the stored one (system prompt or windows.json changed since the call ran).

Migrate a v1 checkout

benchmark migrate-raw

One-time conversion of the old per-call .txt layout to per-model JSONL shards. Verifies every body byte-exact before deleting, backs up calls.jsonl, and keeps any prompt file that does not reconstruct from the corpus. Safe to re-run if interrupted.

Concurrency

benchmark run dispatches calls via asyncio.gather against the OpenAI / Anthropic SDKs. Two semaphores cap concurrency:

• [run] max_concurrent_calls (default 8): global cap.
• [run] max_concurrent_per_provider (default 4): per-provider cap.

Two simultaneous benchmark run invocations against the same calls.jsonl are unsupported and will produce duplicate entries. The runner is single-process by design.

Auth

MinusPod uses Flask sessions. benchmark capture reads the password from MINUSPOD_PASSWORD, logs in once, and caches the cookie at ~/.cache/minuspod-benchmark/session.json (mode 0600). Login is rate-limited to 3/min and 10/hour. Cache TTL is 23 hours. The 429 path is reported but never auto-retried.

Determinism

Every (model, episode, trial, window_index) combination computes a prompt_hash over the system prompt, user prompt, model id, and temperature. The runner skips any tuple whose hash already appears in calls.jsonl. Editing windows (via regenerate-windows --force) changes the hash and forces a re-run for affected windows.

Addressing-mode A/B

MinusPod 2.91.0 added an experimental segment-ID addressing mode alongside the default timestamps mode: instead of [12.3s - 18.0s] text transcript lines and timestamp fields in the model’s JSON, the transcript is a numbered list ([7] text) and the model reports integer start_id/end_id. The benchmark can run either mode, or both, so the two can be compared.

benchmark run                                    # timestamps mode (default)
benchmark run --addressing-mode segment_ids       # segment-ID mode

Both commands append to the same calls.jsonl; each call record carries the addressing_mode it ran under (records written before this field existed are treated as timestamps). benchmark report and benchmark run both accept --addressing-mode and only render calls recorded under that mode, so a report never blends the two: run benchmark report --addressing-mode segment_ids to see the segment-ID side, and plain benchmark report for the timestamps side. If a model ignores the segment-ID contract on a given window, the harness falls back to timestamp parsing for that window and counts it under “ID-contract misses” in that model’s per-model detail section, so a fallback-heavy model is visible rather than silently scored on approximate boundaries.

benchmark show-prompt reconstructs a call’s prompt using the mode stored on that call’s record, so it works unmodified for either mode.

TypeSafe Jev spike (Pass A)

An experimental second detection shape, scored by the same metrics as the chat-model rows so the numbers are directly comparable.

Instead of one call per window returning a JSON array of ad spans, it sends the window transcript as state and asks one Noul per segment (“line L0042 is advertising, not editorial content”). The reply is a probability per segment; spans_from_probabilities recovers ad spans by taking contiguous runs. Because a run is delimited by segment edges, no timestamp is ever generated, and boundaries cannot land off-grid.

benchmark jev-spike --oracle overlap    # offline ceiling, no API key, no cost
benchmark jev-spike --oracle off        # live, replays results/raw/jev_cache.json
benchmark jev-spike --oracle off --enter 0.9 --stay 0.4   # threshold sweep, free

Live probabilities are cached in results/raw/jev_cache.json, keyed by a hash of the request payload, so threshold tuning costs nothing after the first pass and editing a question invalidates its entries rather than silently scoring stale answers.

That file is gitignored: at 650KB of JSON it swamped the diff. Populate it once with TYPESAFE_API_KEY set and every number below reproduces offline from then on:

TYPESAFE_API_KEY=... benchmark jev-spike --oracle off   # ~$0.0296, 171 requests

Rebuilding the multi-pass entries as well (--passes 5) costs about $0.11. The --oracle modes need no key at all.

--oracle substitutes the probabilities a perfect per-segment judge would return, derived from truth.txt. That measures the ceiling of the decomposition itself, separating “can this shape work” from “is the model good enough”. Run it before spending anything on a live sweep.

Current ceiling over the 14-episode corpus: F1 0.991, F0.5 0.996, precision 1.000, recall 0.983, both no-ad controls PASS. For reference the best live chat model in results/report.md is claude-haiku-4-5 at F1 0.920 / F0.5 0.908, so granularity is not the limiting factor at IoU >= 0.5. At stricter matching it becomes one: the same oracle answers score F0.5 0.915 at IoU >= 0.8 and 0.915 at IoU >= 0.9, so 26-second segment edges cost about 0.08 F0.5 once near-exact boundaries are required.

The single miss is ep-tosh-show, where two distinct ad breaks sit 27.7s apart with no speech between them. Nothing in the transcript separates them, so they merge into one span. MAX_RUN_GAP_SECONDS (30s) is the knob: swept against the oracle, precision reaches 1.000 at 30s and is flat from there to unbounded, because speech between two breaks produces its own low-scoring segments and ends the run without help. Only a pure-silence gap can bridge two breaks. Lowering it to 15s recovers that one ad but over-splits real breaks that contain internal silence, costing more precision than it buys recall (mean F0.5 0.925 vs 0.996).

Live results

One pass over the corpus at the tuned defaults (--guidance full --metadata):

	F1	F0.5	Precision	Recall	No-ad controls
Jev (jev-latest)	0.929	0.957	0.979	0.893	PASS / PASS
claude-haiku-4-5	0.920	0.908	0.900	0.946	PASS

Do not read that as Jev beating Haiku. Those numbers come from the same twelve episodes every parameter was chosen on. Cross-validation (below) is the honest view, and the four state variants are not separable at this corpus size. The defensible claim is that Jev is competitive with Haiku at roughly 3% of the cost ($0.0021 against $0.0773 per episode, 36x), with detection tilted the way MinusPod wants: precision 0.979 against recall 0.893, so it under-detects rather than over-detects, and it emits one spurious span where Haiku emits about seven.

Boundaries are the other half of that and they do not tilt the same way. Start bias is -3.96s, meaning a matched span opens before the ad and the cut eats editorial; only the end edge leaves audio in, at -0.89s. Seven of the twelve ad-bearing episodes score 1.000 and none is below 0.75.

Stacked ads and the bridge

ep-drink-champs was the one bad episode (F0.5 0.481) and every error in it came from a single mechanism. It carries the same ad stack three times (Airtasker, then a cross-promo for Decisions Decisions, then a Kiki Palmer episode plug, then OnDeck, then Public), and that cross-promo plays a clip of the show it is advertising. The clip is two people chatting, so Jev scores it 0.05-0.27 – correctly, line by line – and the run severs in the middle of the break. One truth break became two half-spans, each failing IoU 0.5: three breaks lost that way produced 3 FN and 6 FP, which was the entire error budget for the episode.

BRIDGE_SECONDS lets a run cross that much low-scoring speech to rejoin. An empty gap is never bridged: no segment in the gap means silence, and silence between two breaks is exactly what separates them (see ep-tosh-show, where two breaks sit 27.7s apart with nothing between).

The sweep is flat below 25s and flat again from 28s to 60s+, and at 30s it moves exactly one episode:

episode	bridge 0	bridge 30
drink-champs	0.481	1.000
the other eleven	–	unchanged, +0.000 each
both no-ad controls	PASS	PASS

That is the strongest argument for keeping it: it repairs a specific understood failure and is a no-op everywhere else. It is also the weakest part of the evidence, because one episode is the entire case for this parameter. Cross-validating enter/stay/bridge together gives held-out 0.914 against 0.957 in-sample, and the whole gap is the single fold that holds out drink-champs: with it removed the training set has no reason to prefer any bridge, picks 0 on a tie, and then scores 0.709 on the held-out pair where 30s would have scored 0.969. Expect the bridge to help on shows that stack ads this way and to do nothing on shows that do not.

How much of this is overfitting

benchmark jev-cv tunes on all but --fold-size episodes, scores those held out, and repeats until every episode has been held out once. It reads the cache, so it costs nothing.

what gets chosen per fold	in-sample	held-out	optimism
enter / stay only	0.957	0.957	+0.000
state variant and thresholds	0.920	0.831	+0.088

(The thresholds row was re-measured after the _bridged fix below; it is the current number. The variant-selection row predates that fix and should be re-checked once all four variant payloads are in the cache.)

Thresholds generalize. Every one of the six folds independently selected enter 0.95 / stay 0.40 with two episodes removed, and held-out scoring matched in-sample exactly (+0.000 optimism). The optimum is not balanced on any particular episode, so tuning it costs nothing in honesty.

Variant selection does not. Folds picked different winners (full+meta, basic, basic, full, full) and each pick scored worse on its own held-out pair than simply using one fixed configuration would have: -0.083. Choosing among the four variants on twelve episodes is fitting noise, which is what the 0.017 F1 run-to-run spread already implied.

So the shipped configuration is fine to keep – it wins on all four metrics at once and carries strictly more information – but the reason to keep it is not that it measurably beats the alternatives, because it does not. Any future state change needs cross-validation, not a single full-corpus score, before it can claim an improvement.

What goes in state

--guidance full carries MinusPod’s whole rulebook (short brand taglines, platform pre/post-rolls, the produced-vs-organic distinction, the explicit not-an-ad list) rather than the one-paragraph summary. --metadata adds the podcast name, episode title, and synopsis. Both ride in state, which is sent once per request, so together they cost about 800 tokens per window against a per-segment question budget that dwarfs them.

Each variant needs its own thresholds – more guidance shifts the probability distribution, so scoring a new variant against the old enter/stay measures the mismatch rather than the variant:

variant	enter	stay	F1	F0.5	Precision	Recall
basic guidance, no metadata	0.98	0.50	0.883	0.909	0.929	0.851
full guidance	0.97	0.40	0.886	0.905	0.920	0.865
metadata only	0.98	0.50	0.845	0.884	0.919	0.799
full guidance + metadata	0.95	0.40	0.890	0.914	0.934	0.862

Metadata on its own is the worst variant and only helps alongside the full rulebook. Cross-validation says that apparent interaction is noise: these four rows are not separable on twelve episodes, and picking the best-looking one costs 0.083 F0.5 on held-out data. Two runs of one identical configuration differed by 0.017 F1, which is the scale these gaps live at.

These thresholds were fitted on the same 12 episodes they are scored on, so 0.909 is optimistic. Treat it as “worth a real evaluation”, not as a measured production number. Held-out episodes are the next step.

ENTER_THRESHOLD / STAY_THRESHOLD dominate the result and were swept against the cache. The signal is strongly bimodal: across the cached corpus 43% of segments come back at exactly 0.02 and only 4.7% land anywhere in 0.2-0.8, the top bucket is 0.98, and Jev reports two decimals with a maximum of 0.99, so any threshold above 0.99 matches nothing. A run should open only on near-certainty and then extend generously across the weaker shoulders of the same break. Swept on the basic-guidance variant, which shows the shape most clearly (the shipped defaults are the full-guidance row above):

enter	stay	F1	F0.5	Precision	Recall
0.60	0.40	0.731	0.667	0.632	0.903
0.90	0.40	0.778	0.755	0.744	0.834
0.95	0.40	0.827	0.829	0.834	0.834
0.98	0.50	0.883	0.909	0.929	0.851
0.99	0.50	0.697	0.774	0.843	0.610

The no-ad controls only pass from enter 0.70 upward.

Cost of one live pass over the whole corpus is $0.0296 ($0.0021 per episode) at $0.042 per million input tokens with output unbilled, measured from the cached entries: 703,925 input tokens across 171 window requests. The earlier basic-guidance-only pass cost about $0.022 on the same corpus; the difference is the shipped variant’s fuller state. A 10-pass self-consistency sweep is about $0.30. JSON compliance and Extraction methods do not apply here: the response is typed, so there is no parsing step to fail.

Multi-pass (--passes N): tried, buys nothing

--passes N takes N independent draws per window (distinct uid, so each is its own cache entry) and averages them, so a segment the passes disagree on lands mid-scale and falls below enter instead of opening a run on a coin flip. Self-consistency is the headline use for a model this cheap, so it looked like free accuracy.

It is not. Re-tuning thresholds for each N (averaging compresses the distribution, so the single-pass thresholds do not transfer):

passes	enter	stay	F1	F0.5	Precision	Recall	cost
1	0.95	0.50	0.890	0.914	0.934	0.862	$0.022
2	0.95	0.50	0.890	0.914	0.934	0.862	$0.044
3	0.95	0.50	0.890	0.914	0.934	0.862	$0.066
5	0.95	0.50	0.890	0.914	0.934	0.862	$0.111

Identical, to three decimals, at five times the price. (Scored at the default thresholds instead, five passes lose 0.038 F0.5 – that gap is threshold mismatch, not the passes.)

The reason is that Jev is close to deterministic on this workload, more so than the published figures suggest:

	measured	docs
mean per-segment stdev across 5 passes	0.0031	0.0102
mean spread (max - min)	0.0080	–
segments identical across all 5 passes	59.5%	–
segments straddling the 0.95 decision boundary	18 / 4228 (0.4%)	–

Eighteen contested segments corpus-wide, and none of them changes a span decision. There is nothing for averaging to repair. Repeat draws would only earn their cost on a model that actually wavers, or if the spread itself were used as a routing signal (hold the contested spans for review rather than averaging them away) – and 0.4% is too thin a slice to be worth a stage.

Pass B (--confirm): tried, does not help

Pass A opens a run only at enter 0.98, which is doing two jobs at once – “is this an ad” and “am I sure enough to cut”. The obvious fix is to split them: generate candidates at a loose threshold, then confirm each assembled span with a second request (CONFIRM_QUESTIONS: is_ad, promotional_language, produced_insert, guest_own_work, host_organic), with ConfirmPolicy combining them in code.

It does not work, in either arrangement:

Configuration	F1	F0.5	Precision	Recall
Pass A tuned, no confirmation	0.883	0.909	0.929	0.851
Loose candidates (0.60) + best policy	0.814	0.811	0.812	0.834
Tuned candidates (0.98) + any policy	0.883	0.909	0.929	0.851

On tuned candidates confirmation rejects nothing: no min_is_ad between 0 and 0.7 changes a single span, and above that it only loses recall while precision stays pinned at 0.929. On loose candidates it does filter (precision 0.632 -> 0.812) but never recovers the ground given up.

The reason is that the premise was wrong. Pass A’s per-segment Nouls already receive the whole window as state – every question sees every line, they just answer about their own – so a segment was never being judged as an isolated fragment. Re-asking at span level gives the model no information it did not already have. The residual errors are genuine model disagreement, not question framing.

max_exclusion is inert throughout: guest_own_work and host_organic essentially never fire on real candidates, so those two questions earn nothing on this corpus.

The code is kept because the negative result is cheap to re-verify and the ConfirmPolicy shape is the right one if the signals ever become discriminating. Improving accuracy from here means attacking model judgment directly – richer guidance, episode metadata in state – not more stages.

Adding a new model or episode

• New model: append [[models]] to benchmark.toml. benchmark run will fill the gaps (existing models stay cached).
• New episode: capture and verify it. benchmark run will run the configured models against it.

Cost

Costs are recomputed from token counts at report time using the latest pricing snapshot in data/pricing_snapshots/. The *_at_runtime fields in calls.jsonl preserve actual spend.

A full sweep across the recommended 32-model list and 7-episode corpus is roughly $80-$300 depending on model mix. Use --dry-run before kicking off to see the call count.

Schema versions

Every record in calls.jsonl and episode_results.jsonl carries schema_version. Schema changes require a coordinated writer/reader update + version bump.

• v1: one .txt per call under responses/, one .txt per prompt_hash under prompts/; records carry prompt_path.
• v2 (current): response bodies in per-model JSONL shards (responses/<model>.jsonl, keyed by call_id); prompts are not stored (reconstruct via show-prompt); prompt_path dropped and response_path points at the shard. Convert a v1 checkout with benchmark migrate-raw.