LLM benchmark

Offline tool that compares LLMs on ad-detection accuracy, cost, latency, and JSON compliance using real MinusPod transcripts. Produces a Markdown report committed to this repo.

Layout

benchmarks/llm/
  pyproject.toml             # uv project, imports MinusPod modules at runtime
  benchmark.toml.example     # copy to benchmark.toml and fill in (gitignored)
  .env.example               # copy to .env (gitignored)
  src/benchmark/             # source
  data/
    corpus/                  # committed verified episodes
    candidates/              # gitignored work-in-progress captures
    pricing_snapshots/       # committed pricing history
  results/
    raw/                     # calls.jsonl, episode_results.jsonl, responses/<model>.jsonl shards
    report.md                # current report
    report_assets/           # SVG charts referenced by report.md
    archive/                 # explicit snapshots: results/archive/<date>/

Raw response bodies live in one JSONL shard per model (results/raw/responses/<model>.jsonl, lines of {call_id, body}), so the file count stays flat as the corpus grows. Prompts are not stored at all: they reconstruct deterministically from the corpus, and benchmark show-prompt <call_id> verifies the result against the prompt_hash recorded at call time. benchmark show-response <call_id> prints a single raw response body.

Setup

From benchmarks/llm/:

cp benchmark.toml.example benchmark.toml   # edit to fill in your MinusPod base_url
cp .env.example .env                       # fill in MINUSPOD_PASSWORD and provider keys
uv sync                                    # installs deps into benchmarks/llm/.venv

Then run any command via uv run benchmark <cmd>.

The CLI auto-loads benchmarks/llm/.env on startup (resolved relative to the package, not the CWD, so it works from any directory). Shell-exported variables take precedence over .env values.

Requires MinusPod >= 2.0.28 on the server you point at: the benchmark imports create_windows from ad_detector and fetch_litellm_pricing from pricing_fetcher. Before 2.0.28 those modules eagerly imported webhook_service (which pulls jinja2) and bs4, so a clean uv sync could not start the CLI. 2.0.28 deferred both. The benchmark also calls GET /api/v1/feeds/{slug}/episodes/{id}/original-segments, an endpoint added in 2.0.26. Older episodes return 404 until reprocessed.

Transcript source

segments.json for every corpus episode is pulled byte-exact from the source MinusPod instance’s original-segments endpoint. The transcript itself was generated by faster-whisper inside that instance, not by the benchmark. The Whisper config used:

Setting	Value
Model	large-v3
Backend	local (faster-whisper, CUDA GPU)
Compute type	auto (resolves to float16 on CUDA)
Language	en (forced English, not auto-detect)
VAD gap detection	on (start 3.0s / mid 8.0s / tail 3.0s)

The model.transcribe() call shape (from src/transcriber.py):

WhisperModel(model_size="large-v3", device="cuda", compute_type="auto")
model.transcribe(
    audio,
    language="en",
    initial_prompt=<podcast name + SEED_SPONSORS vocabulary>,
    beam_size=5,
    batch_size=<adaptive: 16/12/8/4 by episode length>,
    word_timestamps=True,
    vad_filter=True,
    vad_parameters={"min_silence_duration_ms": 1000, "speech_pad_ms": 600, "threshold": 0.3},
)

This is the call as it stood when the corpus was transcribed. The initial_prompt seeded 26 sponsor terms (AD_VOCABULARY_TERMS, plus the podcast name) so Whisper produced consistent spellings (Athletic Greens rather than AG1, ExpressVPN rather than express vpn). That biased what showed up in the transcript, and therefore what every benchmarked LLM is scored against. 2.96.0 removed the prompt: on the batched pipeline it cost 7-17% of each episode’s speech, so transcripts from current releases have no prompt, fuller coverage, and no spelling bias. The corpus has not been re-transcribed since. Sponsor names are still matched against src/utils/constants.py SEED_SPONSORS (~250 sponsors, each with canonical name, intentional alias spellings, and category).

A second mapping, SPONSOR_ALIASES, is applied after transcription to normalize Whisper mishearings toward the canonical name (a firm -> Affirm, xerox -> Xero, pure tea -> Purity). About 174 such corrections in production. This affects sponsor-name matching but not the transcript fed to the LLM.

Both lists are rendered in full in the generated results/report.md under “Transcript source”. A small canonical-name sample: Athletic Greens, BetterHelp, Squarespace, Shopify, HelloFresh, NordVPN, ExpressVPN, ZipRecruiter, SimpliSafe, Mint Mobile.

Two practical consequences for anyone reading the F1 numbers:

• F1 numbers are upper-bounded by transcript quality. A smaller Whisper model would produce a different (almost certainly worse) ceiling.
• If a model scores well here, it’s scoring against large-v3 transcripts made with the pre-2.96.0 prompt. Deployments running smaller Whisper models should expect lower F1; 2.96.0 and later transcribe without the prompt, which recovers dropped speech (sponsor reads included) but loses the spelling bias, and that combination has not been benchmarked.

Common workflows

Capture a new episode

benchmark capture --episode-url https://podsrv.example.com/ui/feeds/<slug>/episodes/<id>
# edit data/candidates/ep-<slug>-<id>/truth.txt to verify ad markers
benchmark verify ep-<slug>-<id>

Run the benchmark

benchmark refresh-pricing                  # fetch a fresh pricing snapshot
benchmark run                              # auto-fill all gaps, regenerate report
benchmark run --dry-run                    # preview what would run, no API calls
benchmark run --retry-errors               # also retry calls recorded with error

benchmark run always reads [[models]] from benchmark.toml and data/corpus/ for episodes. To restrict scope, edit the config (set deprecated = true) or move episode directories. There are no --model or --episode filters.

Regenerate the report from existing data

benchmark report

Useful after editing the report template. No LLM calls happen.

Snapshot the report

benchmark archive

Copies results/report.md + assets to results/archive/<YYYY-MM-DD>/.

Audit a stored call

benchmark show-response <call_id>          # raw response body from its per-model shard
benchmark show-prompt <call_id>            # rebuild the user prompt from the corpus + verify prompt_hash
benchmark show-prompt <call_id> --snapshot prompts/05-22-206.txt   # verify against a frozen system prompt

show-prompt exits 3 with a MISMATCH warning when the recomputed hash differs from the stored one (system prompt or windows.json changed since the call ran).

Migrate a v1 checkout

benchmark migrate-raw

One-time conversion of the old per-call .txt layout to per-model JSONL shards. Verifies every body byte-exact before deleting, backs up calls.jsonl, and keeps any prompt file that does not reconstruct from the corpus. Safe to re-run if interrupted.

Concurrency

benchmark run dispatches calls via asyncio.gather against the OpenAI / Anthropic SDKs. Two semaphores cap concurrency:

• [run] max_concurrent_calls (default 8): global cap.
• [run] max_concurrent_per_provider (default 4): per-provider cap.

Two simultaneous benchmark run invocations against the same calls.jsonl are unsupported and will produce duplicate entries. The runner is single-process by design.

Auth

MinusPod uses Flask sessions. benchmark capture reads the password from MINUSPOD_PASSWORD, logs in once, and caches the cookie at ~/.cache/minuspod-benchmark/session.json (mode 0600). Login is rate-limited to 3/min and 10/hour. Cache TTL is 23 hours. The 429 path is reported but never auto-retried.

Determinism

Every (model, episode, trial, window_index) combination computes a prompt_hash over the system prompt, user prompt, model id, and temperature. The runner skips any tuple whose hash already appears in calls.jsonl. Editing windows (via regenerate-windows --force) changes the hash and forces a re-run for affected windows.

Addressing-mode A/B

MinusPod 2.91.0 added an experimental segment-ID addressing mode alongside the default timestamps mode: instead of [12.3s - 18.0s] text transcript lines and timestamp fields in the model’s JSON, the transcript is a numbered list ([7] text) and the model reports integer start_id/end_id. The benchmark can run either mode, or both, so the two can be compared.

benchmark run                                    # timestamps mode (default)
benchmark run --addressing-mode segment_ids       # segment-ID mode

Both commands append to the same calls.jsonl; each call record carries the addressing_mode it ran under (records written before this field existed are treated as timestamps). benchmark report and benchmark run both accept --addressing-mode and only render calls recorded under that mode, so a report never blends the two: run benchmark report --addressing-mode segment_ids to see the segment-ID side, and plain benchmark report for the timestamps side. If a model ignores the segment-ID contract on a given window, the harness falls back to timestamp parsing for that window and counts it under “ID-contract misses” in that model’s per-model detail section, so a fallback-heavy model is visible rather than silently scored on approximate boundaries.

benchmark show-prompt reconstructs a call’s prompt using the mode stored on that call’s record, so it works unmodified for either mode.

Adding a new model or episode

• New model: append [[models]] to benchmark.toml. benchmark run will fill the gaps (existing models stay cached).
• New episode: capture and verify it. benchmark run will run the configured models against it.

Cost

Costs are recomputed from token counts at report time using the latest pricing snapshot in data/pricing_snapshots/. The *_at_runtime fields in calls.jsonl preserve actual spend.

A full sweep across the recommended 32-model list and 7-episode corpus is roughly $80-$300 depending on model mix. Use --dry-run before kicking off to see the call count.

Schema versions

Every record in calls.jsonl and episode_results.jsonl carries schema_version. Schema changes require a coordinated writer/reader update + version bump.

• v1: one .txt per call under responses/, one .txt per prompt_hash under prompts/; records carry prompt_path.
• v2 (current): response bodies in per-model JSONL shards (responses/<model>.jsonl, keyed by call_id); prompts are not stored (reconstruct via show-prompt); prompt_path dropped and response_path points at the shard. Convert a v1 checkout with benchmark migrate-raw.