Jev benchmark audit, 2026-09-19

Scope: independent verification of the Jev-vs-chat-model comparison, prompted by an external first-pass review. Seven parallel investigations against the real code and the 171-window probability cache. No API calls except the live ablation noted below.

Every number here was reproduced from src/benchmark/ and results/raw/, not taken from the review or from results/report-combined.md.

1. Verdict

The headline “Jev beats chat LLMs at ad detection” is not supported.

comparison	jev F0.5	haiku F0.5	diff	t(11)
as published, jev tuned in-sample, 1 pass	0.9572	0.9077	+0.0495	+1.887
jev cross-validated over its real selection surface	0.9139	0.9077	+0.0063	+0.131
jev measured at 5 passes, shipped thresholds	0.9190	0.9077	+0.0113	see 2a

One-sided critical value at 11 df is 1.796. The published version clears it by 0.09. Two-sided it never clears. After cross-validating the parameters chosen by looking at this corpus, the difference is +0.006 F0.5 on 12 episodes with a paired standard error of 0.048.

README.md:245 already states this in prose: “Do not read that as Jev beating Haiku.” The generated table in results/report-combined.md contradicts its own README.

The honest claim, which the evidence does support:

On a 14-episode corpus, Jev’s per-segment decomposition is statistically indistinguishable from claude-haiku-4-5 at roughly 1/37 the cost and 1/59 the latency, and never cuts audio that contains no advertising. It removes about 3x more editorial content than Haiku does, because segment quantization makes every boundary error cost a whole 26-second segment.

2. The lead does not survive strict matching

The entire ranking is computed at IoU 0.5 (report/aggregate.py, DEFAULT_IOU_THRESHOLD).

IoU	jev F0.5	rank among chat models	chat models above jev
0.50	0.9572	1	0
0.70	0.8749	1	0
0.76	0.8332	1	0
0.78	0.7811	2	1
0.80	0.7484	5	4
0.90	0.7156	4	3

The lead is lost between IoU 0.76 and 0.78. First model past it is qwen/qwen3.5-plus-02-15. At IoU 0.8, four chat models beat jev.

Jev’s drop from IoU 0.5 to 0.8 is -0.2088, the largest of any model in the top 25. It is the steepest beneficiary of the 0.5 threshold in the entire benchmark.

2a. Self-consistency makes jev worse, not better

Measured 2026-09-19, benchmark jev-spike --oracle off --passes 5, 855 fresh windows.

	F1	F0.5	P	R
1 pass (published)	0.9294	0.9572	0.9792	0.8931
5 passes, same thresholds	0.8960	0.9190	0.9380	0.8650

aggregate_passes means the per-segment probabilities before assembly. The single-pass distribution is sharply bimodal (3839 of 4228 answers below 0.40, 210 at or above 0.95), so averaging moves mass toward the middle and fewer segments clear the 0.95 enter gate. Precision falls 0.979 to 0.938, recall 0.893 to 0.865.

ENTER_THRESHOLD was fitted to the single-pass distribution and does not transfer to averaged probabilities. This is a ninth entry for the selection surface in section 7: the value of --passes interacts with the thresholds, and README.md:380-395 records the sweep as “all identical”, which no longer holds.

Deliberately NOT re-tuned. Sweeping enter against the averaged probabilities on the same 12 episodes would reopen exactly the optimism problem this audit documents. The constants stay frozen and the result is reported as measured.

Note that 0.9190 lands next to the cross-validated 0.9139, which is mild corroboration that 0.9572 was the optimistic number rather than the representative one.

3. At IoU 0.5 the margin is inside the noise

Bootstrap, 10,000 draws, resampling the 12 ad-bearing episodes with replacement, paired.

comparison	IoU	diff	P(jev higher)	95% interval
jev vs haiku[segment_ids] trial 1	0.5	+0.0056	0.601	[-0.0283, +0.0438]
jev vs haiku[timestamps], as published	0.5	+0.0495	0.979	[+0.0021, +0.1000]
jev vs haiku[timestamps]	0.8	-0.0119	0.454	[-0.1475, +0.1098]
jev vs qwen3.5-plus	0.8	-0.0441	0.234	[-0.1666, +0.0680]

The published interval excludes zero by two thousandths of a point. Per-episode record is 6-2-4: jev ties a third of the corpus outright.

Jev is one deterministic pass. Haiku’s trial-to-trial range in segment_ids mode is 0.1058 F0.5, more than double jev’s entire 0.0495 headline margin. Jev’s number is a single draw compared against a distribution wider than the effect being claimed.

4. What jev actually is

Per-trial means summed over the 12 ad episodes, 49 canonical truth spans.

model	pred spans	missed	spurious	sloppy (hit@0.5, miss@0.8)	sloppy %
jev-ceiling (oracle)	48.0	1.0	0.0	3.0	6%
jev	45.0	5.0	1.0	11.0	25%
claude-haiku-4-5	52.6	3.2	6.8	7.6	17%
qwen3.5-plus-02-15	52.0	4.4	7.4	5.4	12%
google/gemini-3.5-flash	54.2	4.0	9.2	6.8	15%

Jev emits 1 spurious span where the chat models emit 6 to 10. That single number buys the 0.9572, because F0.5 weights precision 2x and IoU 0.5 prices boundary error at zero.

But jev also misses more ads than Haiku (5 of 49 vs 3.2) and cuts the ones it finds more loosely than anything else in the top 13 except grok-4.3.

So the profile is not “better”. It is different: jev almost never hallucinates an ad, at the cost of missing more of them and cutting all of them sloppily. Whether that wins is a product decision about the price of a false positive against a loose boundary. The benchmark’s metric currently makes that decision silently.

5. Error decomposition

A three-level ladder separating granularity from assembly from judgment. The “ceiling” row in the existing report cannot do this, because it runs the same assembly code as the live row and only swaps the probabilities.

L0 truth -> L1 truth snapped to segment edges -> L2 oracle labels through the real assembler -> L3 live jev.

At IoU 0.8, jev’s 0.2781 F1 shortfall from perfect splits:

component	F1 delta	share
granularity (L0 to L1)	0.0583	21.0%
assembly (L1 to L2)	0.0296	10.7%
judgment (L2 to L3)	0.1901	68.4%

Judgment costs 6.4x what assembly does. The entire assembly delta is one episode: span lists are byte-identical on 11 of 12, and only ep-tosh-show diverges.

That assembly residual is irreducible, not a tuning failure. max_gap swept from 0 to 90s in 0.1s steps never reproduces L1, because the corpus contains a direct contradiction:

• ep-security-now has a 28.93s pure-silence wall INSIDE one truth ad (needs gap >= 28.93)
• ep-tosh-show has a 27.72s pure-silence wall BETWEEN two distinct ads (needs gap < 27.72)

_bridged and the 15s canonicalization gap contribute exactly zero at every value tested.

Caveat on the granularity row: the corpus truth is authored on the segment grid. 53 of 53 raw truth ad starts land exactly on a segment start. The 21% granularity share is a floor; against independently annotated truth it would be materially larger.

Judgment failure anatomy

• Spurious spans: 0. Every one of jev’s 45 spans lands on a real ad. All 12 “false positives” at IoU 0.8 are overlapping spans that fail the threshold, not hallucinations.
• Missed entirely: 4 of 49. Three are episode-opening pre-rolls.
• Boundary errors are bimodal, not Gaussian: 36 of 45 starts and 36 of 45 ends are exact within 1s; the other 9 are off by 5 to 72s. The failure mode is “one shoulder segment got the wrong label”, not gradual drift.
• Concentration: drink-champs alone carries 27.4% of judgment loss, the top 3 episodes carry 71.2%, and 4 of 12 episodes contribute zero or negative.

6. Confirmed bug: span truncation at jev.py:351

spans_from_probabilities sets a span’s end to members[-1]["end"], the last member’s end, rather than the maximum end over members. Because jev.py:323 sorts by start, a segment that strictly contains the following segment truncates the span.

Live case, ep-security-now-audio-2850b24903b2, segments 108/109:

	span
as shipped	2560.07 - 2664.88
with max-end	2560.07 - 2687.39

Truth is 2560.07 - 2693.01. 22.51s of a real ad break left in; that episode’s IoU goes 0.7884 -> 0.9577.

Corpus-wide impact of the fix: neutral at IoU 0.5/0.6/0.7, +0.0119 F1 and F0.5 at IoU 0.8/0.9, end MAE -0.268s at IoU 0.5. Small because the pattern only bites when the contained segment is a run’s last member, which happens once in 45 spans.

It should land on correctness grounds regardless. There are 15 strict superset/subset pairs corpus-wide and the largest truncation available is 27.11s (tim-dillon sids 114/113).

The analogous stale-frontier forms at jev.py:339 and jev.py:379 change nothing on this corpus but have the same shape and were fixed for symmetry.

Status: fixed 2026-09-19

All three forms corrected in spans_from_probabilities and _bridged: the span end is now the furthest end over run members, the max_gap comparison tracks a running frontier, and the bridge wall measures from the previous run’s furthest end. Four tests added as tests/test_jev.py::TestContainedSegments; reverting any one of the three fixes fails exactly one of them.

Suite 285 -> 289 passed. Measured effect, single pass:

IoU	F1 before	F1 after	F0.5 before	F0.5 after
0.5	0.9294	0.9294	0.9572	0.9572
0.7	0.8484	0.8484	0.8749	0.8749
0.8	0.7219	0.7338	0.7484	0.7603
0.9	0.6892	0.7011	0.7156	0.7275

Both controls still PASS. Worth noting: post-fix jev at IoU 0.8 scores F0.5 0.7603, which is claude-haiku-4-5’s IoU-0.8 figure to four decimals. A dead heat at strict matching.

Root cause of the duplicates is upstream, not the benchmark: src/transcriber.py:353 merge_overlapping_segments keeps a new-chunk segment unconditionally when it starts inside the 30s overlap but extends past it. The overlaps are periodic at 1318s spacing, exactly chunk duration minus CHUNK_OVERLAP_SECONDS.

7. The selection surface

Every constant whose value the README justifies by a sweep against these 14 episodes.

#	constant	file:line	value	in published CV?
1	ENTER_THRESHOLD	jev.py:43	0.95	yes
2	STAY_THRESHOLD	jev.py:44	0.40	yes
3	MAX_RUN_GAP_SECONDS	jev.py:50	30.0	no
4	BRIDGE_SECONDS	jev.py:59	30.0	no
5	GUIDANCE_FULL vs GUIDANCE	jev.py:85-115	full	no
6	--metadata on/off	jev.py:163-171	on	no
7	_bridged empty-gap rule	jev.py:355-379	gap holds >=1 segment	no
8	run-must-contain-opener rule	jev.py:335-338	required	no, and no sweep cited

The published CV covers 2 of 8.

Expanded leave-2-out CV, same folds, same cache:

surface	grid	in-sample	held-out	optimism
enter/stay (published)	35	0.957	0.957	+0.000
enter/stay/max_gap	245	0.957	0.957	+0.000
enter/stay/bridge	175	0.957	0.914	+0.043
all four	1225	0.957	0.914	+0.043

The +0.000 claim at README.md:299 is true and reproduces exactly. It is also close to vacuous: 35 grid points, one clear optimum, and a sharply bimodal probability distribution (3839 of 4228 answers below 0.40, 210 at or above 0.95). Thresholds do not overfit because the signal is bimodal, not because the pipeline was validated.

MAX_RUN_GAP_SECONDS genuinely adds no optimism. BRIDGE_SECONDS adds +0.043, all from one fold. That figure is already in README.md:285-289; it is absent from the summary table at :297-300 that a reader takes away.

The state-variant dimension (rows 5-6) is not recomputable from the current cache, which holds only the full+metadata payloads.

The asymmetry

Jev had 8 configuration choices fitted against these 12 episodes, 2 of them cross-validated. The chat models had zero: git log -- prompts/ shows one commit (02b90d8f, 2026-08-13), never edited. No content retries, no model-specific prompt variants, no model branching anywhere in src/ad_detector/.

8. Confounds that do NOT exist

Three things that look like they should be unfair to jev, and are not.

Granularity handicap. 93.3% of chat-model start boundaries and 89.3% of end boundaries already sit within 0.05s of a segment edge, because the prompt renders segments as [{start:.1f}s - {end:.1f}s] and the models copy the numbers back. Snapping every chat prediction to jev’s grid moves mean F0.5 by +0.0004 at IoU 0.5 and -0.0003 at IoU 0.8. Both sides already work on the same grid, and jev’s boundaries are the worse ones.

Parse failures. claude-haiku-4-5 had 0 failed windows out of 855. Excluding parse-failure windows changes the top 5 chat models’ F0.5 by +0.000 each. The confound is devastating at the bottom of the table (bytedance-seed/seed-2-1-turbo loses 99.0% of ad opportunities) but absent at the top.

Trial-count fairness. _recompute_costs runs per (model, episode, trial) and averages over trials before summing, so the published chat cost is already one-pass-equivalent.

Two confounds run the other way, in the chat models’ favor:

• The chat prompt ships a 296-brand sponsor gazetteer. 42 of those brands appear verbatim inside this corpus’s truth spans, including 12 in ep-security-now.
• The chat side is asked a strictly harder question (find the ad, name its category, transcribe boundary timestamps, emit schema-valid JSON) and scored on part of it.

9. Reporting defects

Three defects previously believed fixed have not landed.

Cost normalization is still split. jev.py:820 divides by episode count; aggregate.py:420-422 accumulates a corpus total. Same column, two denominators, 14x apart. A report regenerated today prints jev at $0.0021 against haiku at $1.0820, a 515x ratio. The true like-for-like figure is 36.6x ($0.029565 vs $1.0820 corpus-wide, or $0.002112 vs $0.07729 per episode).

README.md:250-251 claims “roughly 1% of the cost”. Actual is 2.73%.

The column label at report-combined.md:31 (“Average dollars per episode”) is wrong for every row, not just jev. Divide every published figure by 14.

results/report-combined.md is internally inconsistent mid-migration: $0.0296 at lines 59, 152, 1448, 1556 and $0.0021 at line 982, same row, same file.

Latency still publishes 0.0s. All 171 cache entries have elapsed_ms == 0. The guard at jev.py:840 is correct and writes nothing, but ModelStats.p50_call_latency_ms defaults to 0.0 (aggregate.py:103-107) and sections.py:104 / sections.py:815-816 render it unconditionally. There is no “unavailable” path. Net effect is identical to the hardcoded zero that was thought to be removed: jev is published as the fastest system in the benchmark, ahead of google/gemini-3.5-flash-lite at 0.7s, and the latency-tail table sorts by p50 so jev sits at the top of it. Affected: report-combined.md:59, :713, :714, :1448, :1556-1557.

Determinism published 0.0000 from n=1. Corrected on inspection: the current code already skips a row with no f1_stdev_per_episode, so the jev / jev-ceiling 0.0000 rows at report-combined.md:991-992 are stale output from an earlier version. The defect is real but differently shaped: metrics.trial_stdev returns 0.0 for n < 2 (metrics.py:303-305), so any single-trial row publishes a fabricated 0.0000 as if measured. That table also never sorted by stdev; it sorts by mean F1 (sections.py:881), so jev led it on accuracy, not on determinism.

Pricing provenance. INPUT_COST_PER_MTOK = 0.042 (jev.py:30-31) is sourced to a URL in a comment, with no committed snapshot, unlike every chat model. Output cost is unmodeled; jev produced 79,098 output tokens (11.2% of its input). Even at 10x input pricing jev stays 17x cheaper, so the cost advantage is real. Only its magnitude is wrong.

Provenance. results/report-combined.md is untracked and src/benchmark/jev.py has no git history at all. No constant’s origin can be traced through commits.

Doc error. report.md:45 and report-combined.md:45 say episodes are split into “~85-second” windows. Measured lengths are 410-595s, mean ~550s. Off by about 6.5x in the section explaining the methodology.

10. The external review, point by point

claim	verdict
Baseline reproduces (F1 0.929, F0.5 0.957, P 0.979, R 0.893)	confirmed exactly
IoU sensitivity table	numbers confirmed
“Falsifies the ceiling framing”	no. README.md:221-226 already scopes the claim to IoU >= 0.5 and documents the strict-IoU cost
“The decomposition is fine; the assembly is not”	does not follow. Both rows share the same assembly code. Measured split is 10.7% assembly, 68.4% judgment
Overlap oracle is not inflating the ceiling	confirmed at IoU 0.5. At 0.8 overlap slightly DEFLATES it (0.912 vs majority 0.931)
Start bias -3.96s	confirmed, but it is 5 spans across 4 episodes, and 1.19s of it is quantization the oracle shares
lead=0.50 is “strictly free”	no. +29.1s ad left in. A 3.4:1 trade, not free
lead=0.90 is the right value	no. Exchange rate past 0.50 is 1.05:1, and 0.90 inverts the bias (-3.96s to +1.79s) rather than removing it
Leave-2-out CV on lead	confirmed, and honest. The fold-2 tie breaks against their own thesis
“Detection untouched”	holds for lead 0.40-0.90, breaks at 0.95 (45 -> 46 spans). Not a structural guarantee
TAIL_THRESHOLD = 0.40	correct value, dead code. Run membership already requires >= stay, so the tail loop can never trim
Duplicate segments exist	confirmed, 63 overlapping pairs, 12 of 14 episodes
“Sorting changes which duplicate lands in the window”	no. Member sets are identical in all 171 windows. 22 reorder, 24 cache keys change
“A duplicate pair straddling an ad boundary”	no. Zero corpus-wide
The patch applies	no. Bare @@ headers; neither git apply nor patch accepts it
Open question: does Haiku show the same outward edge?	no. +0.87s at IoU 0.5, inward

The review’s most valuable contribution is the one it did not name: gesturing at the duplicate-segment problem led to the confirmed truncation bug in section 6.

Why the proposed patch should not land as written

Beyond the value of lead:

1. lead > enter is unguarded. When no member reaches lead, the span collapses to the run’s final and weakest segment. 10 of the 35 points in THRESHOLD_GRID (jev.py:661-665) have enter < lead, so tune_thresholds and cross_validate would silently evaluate a third of the grid under a broken span builder. lead=1.0 collapses every span in the corpus with no error.
2. TAIL_THRESHOLD = 0.40 cannot trim anything (see above).
3. Trimming defeats canonicalize_ads. Shortening spans grows the gaps between them, so pairs that used to merge under the 15s rule no longer do, turning 1 TP into 1 TP + 1 FP.
4. Cross-window dedup undoes the trim. canonicalize_ads takes the union and the earliest start wins, so whenever one window trims less than the other, the least-trimmed boundary is what gets scored.
5. lead is reachable only as a module constant. It is not plumbed through score_episode, merge_into_stats, tune_thresholds, cross_validate, or the CLI. The CV numbers in the patch comment cannot have been produced by the patched code.
6. The test suite is 285 pass before and after. tests/test_jev.py:20-26 pins ENTER, STAY = 0.60, 0.40, so every span test runs in the broken lead > enter regime and none catches it.

10a. Decomposition ablation: the leading edge is the model

Run 2026-09-19. Haiku scored per segment over the same 171 windows from jev.episode_windows, with GUIDANCE_FULL and NOUL_INSTRUCTIONS verbatim, assembled by the unmodified jev.spans_from_probabilities at enter 0.95 / stay 0.40. One deterministic pass, 4228 segment answers, zero missing sids.

row	model	decomposition	F0.5 @0.5	F0.5 @0.8	start bias @0.5
jev	jev	per-segment + assembler	0.9572	0.7603	-3.96s
haiku-per-segment	haiku	per-segment + assembler	0.9160	0.8261	-0.07s
haiku-segment_ids	haiku	span emission	0.9050	0.7221	+0.87s
jev-ceiling	oracle	per-segment + assembler	0.9960	0.9147	-1.19s

Swapping the model under a fixed decomposition moves the start edge 3.89s. Swapping the decomposition under a fixed model moves it 0.94s. The decomposition’s own floor is one episode: tosh-show, where the truth start falls mid-segment and no per-segment method can cut inside it. Excluding it, the oracle’s start bias is exactly 0.00s, jev’s is -3.02s and haiku-per-segment’s is +1.22s.

Mechanism. Of the 179 window-instances jev scores in the 0.40-0.95 band, Haiku pushes 70 above 0.95 and 75 below 0.40, keeping 34. On the segment immediately preceding an ad start, jev has 7 in 0.40-0.95 (4 in 0.40-0.60) against Haiku’s 2. A segment at 0.45 cannot open a run, but STAY 0.40 lets an open run extend backwards across it. The two models agree on the ad body to three decimals (0.870 vs 0.869 at the boundary segment) and diverge only on the shoulder: jev hedges there, Haiku either commits or collapses to 0.01.

Consequence for section 10. The external review’s lead threshold addresses a real symptom in the wrong layer. The mushy shoulder band is a property of jev’s scoring, not of asking one question per segment and gluing runs together, so a second assembly threshold compensates for the judge instead of fixing it.

Consequence for section 1. At IoU 0.8 haiku-per-segment is the strongest non-oracle row measured, above jev (0.7603) and above Haiku’s own span emission (0.7221). The decomposition is not what limits this system; the judge is. Running a chat model through jev’s decomposition beats both jev and that same chat model emitting spans directly.

Cost: $0.7536 notional for the corpus against jev’s $0.0296, about 18x, since jev bills input only at $0.042/Mtok and does not bill output. The accuracy gain is real and it is not free.

Operational note: every failed call returned HTTP 504 from openresty at exactly 90.0s elapsed, against a 180s client timeout in benchmark.toml. The self-hosted wrapper has a hard gateway timeout below the client’s. p50 was 39.2s with a p90 of 67.9s, so the tail runs into that wall. Raise the proxy timeout or drop concurrency before rerunning.

10b. Seconds of audio damage

Threshold-free metric measuring the product’s actual harm: seconds of editorial removed and seconds of ad retained. No IoU anywhere. 12 ad-bearing episodes, 17.484 h of audio, chat rows averaged over 5 trials so every figure is one-pass-equivalent.

row	editorial removed /h	spurious /h	overhang /h	ad retained /h	events >30s /h	pct of damage in >30s
jev 1 pass	14.6s	0.0s	14.6s	21.3s	0.23	62.9%
jev 5 passes	16.8s	0.0s	16.8s	21.3s	0.29	67.7%
jev-ceiling (oracle)	3.6s	0.0s	3.6s	0.0s	0.00	0%
haiku-per-segment	7.2s	2.8s	4.5s	22.9s	0.00	0%
claude-haiku-4-5 [timestamps]	4.7s	0.9s	3.8s	22.6s	0.00	0%
qwen3.5-plus-02-15	19.2s	5.0s	14.2s	17.5s	0.15	39.5%
google/gemini-3.5-flash	14.0s	4.1s	9.9s	19.5s	0.05	13.9%

All 12 rows predicted zero spans on both no-ad controls in every trial, so controls do not separate anything here.

Two findings, pointing opposite ways.

jev never cuts blind. Its spurious damage is exactly 0.0 s/h: across 17.5 hours it never removed a stretch of audio containing no advertising. Every wrongly removed second was attached to a real ad break. Every chat row sits at 0.9 to 5.0 s/h of fully hallucinated cuts, including the haiku port of jev’s own prompt shape at 2.8 s/h. The segment-edge construction is what buys this.

jev’s errors are nonetheless the worse shape. It removes 14.6 s/h of editorial against claude-haiku-4-5’s 4.7 s/h, roughly 3x, and 62.9% of that damage sits in events longer than 30 seconds. jev’s median editorial event is 27.3s; the chat models’ median audible event is 0 to 2s. Both haiku rows produce zero >30s events. Paired bootstrap over 10,000 resamples: P(jev does less damage) = 0.005 against haiku[timestamps] and 0.024 against haiku-per-segment, both intervals excluding zero. jev is Pareto-dominated on the editorial/ad trade under both pooled and macro normalization.

The mechanism is quantization. Because boundaries snap to ~26s segments, every boundary error costs a whole segment. A chat model can be wrong by 2 seconds; jev cannot. So the intuition that a hallucinated span is catastrophic and a boundary error is cheap inverts here: jev is the few-large-errors system and the chat models are the many-small-errors systems. For audio that gets physically cut, many-small is preferable, since a 27s error deletes a paragraph of the show and a 2s error clips a word.

About a quarter of jev’s editorial damage is a floor it cannot cross: jev-ceiling, a perfect per-segment judge through the same assembler, still leaks 3.6 s/h of overhang and 62.8s corpus-wide, purely from segment granularity.

The trade stated in units, per episode: jev costs $0.075 less and finishes about 285s sooner, removes 14.5 more seconds of the listener’s show (21.3s against 6.8s), and leaves about the same amount of ad in (21.3 against 22.6 s/h). That is a cost-and-latency-for- quality trade, not a free win.

Also settles the multi-pass question. jev 1 pass and 5 passes differ in exactly one span corpus-wide: on ep-the-brilliant-idiots the 5-pass version extends 38.3s earlier into editorial. That one span is the entire F0.5 drop from 0.9572 to 0.9193, and ad retained is identical between them. Single pass is strictly better.

Cache preserved at results/raw/haiku_persegment_cache.json (171 windows, 4228 segment answers) so the ablation and this metric are reproducible without re-spending.

10c. Tail risk, not means

The 10b figures are means over 5 trials. Means are the wrong summary for a product whose risk is borne one listener at a time. Recomputed treating each (episode, trial) as one draw, 60 draws per chat row and 60 for jev scored as five independent passes.

row	mean s/h	p90 s/h	worst draw s/h	largest cut s	draws with a >30s cut	with a >60s cut
jev (5 passes as draws)	24.9	65.0	104.5	46.5	18/60	0/60
claude-haiku-4-5 [timestamps]	5.8	18.0	41.1	26.5	0/60	0/60
haiku-per-segment	7.9	25.6	40.2	27.7	0/12	0/12
qwen3.5-plus	24.3	63.4	72.9	70.1	9/60	5/60
grok-4.3	18.5	60.7	146.7	69.9	17/60	5/60
gemini-3.5-flash-lite	8.6	18.0	100.2	133.7	2/60	1/60
sonnet-4-6	18.7	58.8	60.7	64.4	9/60	2/60

Chat models do cut show content, and the worst single event in the study is theirs: gemini-3.5-flash-lite removed 133.7s contiguously on ep-the-tim-dillon-show trial 3 (2286.3-2420.0s), 126.0s of it audible speech, 2.9x jev’s worst. Across all rows there are 65 cuts over 30s, from 7 of 8 chat rows, and 5 of 8 produced a cut over 60s. jev produced none over 60s in 60 draws. Restricted to audible speech, jev’s worst cut anywhere is 35.0s and five chat rows exceed it.

Distribution shape, largest cut over median cut: jev 1.7, haiku[timestamps] 883, grok-4.3 2331, gemini-3.5-flash-lite 4457. jev has no tail; it is the 20-45s bucket. The chat rows concentrate near zero with an open tail that occasionally fires, so a low mean can conceal a draw that deletes two minutes of the show.

Correction to section 10b’s framing: jev’s damage is NOT bounded by construction. The assembler chains segments without a cap, extending across any neighbour above STAY within MAX_RUN_GAP_SECONDS and bridging a further BRIDGE_SECONDS. jev’s 46.5s cut spans two segments in an episode whose largest segment is 29.5s, so the one-segment bound is already broken. What jev has is quantisation, not boundedness.

None of this rescues jev against the actual comparator. claude-haiku-4-5 [timestamps] produced zero >30s cuts in all 60 draws across 175 cuts, largest 26.5s, and wins every risk framing computed: mean, median, p90, CVaR at the 90th percentile (31.7 vs 91.3 s/h), P(draw exceeds 30s or 60s of total removal), adversarial-episode damage, and smallest worst cut. A structural zero carries no sampling variance, which is why the episode bootstrap backs that contrast at P = 0.0000.

Identifiability, stated plainly: 12 episodes cannot identify a tail. Every p90 confidence interval is wider than its own point estimate, and the max statistic is pinned to the observation by construction. gemini-3.5-flash-lite’s 133.7s cut has a bootstrap CI of [16.4, 133.7]s, resting on one episode appearing in the resample. Any claim of the form “model X’s worst case is N seconds” from this corpus describes one or two events, not the model. The only tail claims that should carry weight in a decision are the structural zeros and the >30s-cut rate contrast, CI [+0.067, +0.550].

11. What would settle it

The corpus cannot resolve a difference of this size. 12 ad-bearing episodes, 49 truth spans, paired SE 0.048.

1. A held-out corpus: 10 or more ad-bearing episodes captured after the constants were frozen, scored at the frozen 0.95/0.40/30/30 with no re-tuning. At $0.0021 per episode of jev spend this is a cheap experiment and it is the only thing that would settle the question.
2. BRIDGE_SECONDS and the state-variant dimension inside the CV, with all four variant payloads cached.
3. Real elapsed_ms in the jev cache, so the latency column is a measurement.
4. Either multiple jev trials or a blanked determinism cell, so 0.0000 is not a default rendered as a result.
5. One normalization for the cost column, applied to both sides.
6. A committed pricing source for $0.042/Mtok and a stated basis for “output is not billed”.

11a. Fixed 2026-09-19

defect	change	status
span truncation on contained segments	jev.py span end, max_gap frontier, bridge wall	fixed, 4 tests
cost normalization split 14x	aggregate.py divides by contributing episodes; cost_episodes on ModelStats	fixed
latency default rendered as 0.0s	sections.py _fmt_latency, n/a when no timings, untimed rows sort last	fixed
determinism fabricated from n<2	aggregate.py only writes stdev when trials > 1; n/a rendering	fixed
5-pass data unreachable by the report	passes threaded through merge_into_stats, build_report, --jev-passes	fixed

| ranking readable only at IoU 0.5 | F0.5 @0.8 column on the ranked tables, STRICT_IOU_THRESHOLD in aggregate.py | added |

Suite 285 -> 298 passed. Two pre-existing tests asserted the old table header and were updated; both are column-layout assertions, and no test asserting a value changed.

The strict column keeps the IoU-0.5 sort, so jev still appears second, but now reads 0.957 | 0.760 beside claude-haiku-4-5’s 0.908 | 0.760. The rank inversion described in section 2 is visible from the numbers without re-sorting the table.

Corrected cost, per episode, one pass: jev $0.002112 against claude-haiku-4-5 $0.077287, a ratio of 36.60x. Not the 515x a pre-fix regeneration would have printed, not the 100x README.md:250-251 claims.

With --jev-passes 5 the report now carries measured values instead of defaults:

	1 pass	5 passes
F0.5	0.9572	0.9193
p50 / p95 call latency	n/a	410ms / 489ms
cost per episode	$0.002112	$0.010585
mean F1 stdev across episodes	n/a	0.0242

jev’s 0.0242 is genuinely the lowest trial variance in the benchmark, against claude-haiku-4-5 at 0.0375 and qwen3.5-plus at 0.0264. It had never been measured before, only defaulted to zero.

Note the five-pass configuration is 7.3x cheaper than Haiku rather than 36.6x, and scores lower. Single pass is the better configuration on both axes, and is what the published row should stay on.

12. Method

Seven independent investigations, each re-deriving its numbers from src/benchmark/ and results/raw/ rather than from the published report or the external review. Harness self-checks required each to reproduce jev F1 0.9294 / F0.5 0.9572 at IoU 0.5 and the report’s IoU-0.5 rows before any new result was accepted; the ranking rebuild matched report-combined.md on 86 of 86 rows exactly.

Two corrections surfaced between investigations and are reflected above: strict superset/subset pairs number 15, not 19; and jev’s F0.5 at IoU 0.8 is 0.7484, not 0.7482.

Boundary MAE and bias computed at high IoU are survivor-biased, since only pairs matching at that threshold contribute. Only the IoU 0.5 boundary row is comparable across variants.