cull-audit

cull-audit · public demo study

A top-scoring pick became a reject.

A local-first verification harness for records from AI photo-culling pipelines: bring JSON/JSONL judgments and it writes a flip table, repeat-stability classes, and cost evidence to audit.json and report.md. The study below audits one 67-call reference run on an openly licensed photo set — an example produced with the optional runner.

Read the study ↓ Get the tool ↗
77 photos 67 provider calls 135 records 21 paired 6 flips (all downward) 8/8 repeat groups robust $0.2737 estimated

01 / premise

A score is a lead, not a verdict.

AI culling pipelines often expose scores and rankings that invite more confidence than a single read warrants. cull-audit asks a narrower, answerable question: what do the records actually say — and what happens when the same question is asked again, alone and at higher resolution?

It audits what a culler's records say about stage disagreement, repeat stability, and cost. It never claims human ground truth.

Follow the receipts ↓
Diagram of producer records flowing through validation and audit into evidence files

Scroll horizontally to see the full diagram ↗

02 / protocol

What was run

frozen reference run

What this run found

  • 6 of 21 paired finalists flipped downward
  • 8/8 eligible repeat groups robust
  • $0.273687 estimated across 67 calls

77 openly licensed photos (Wikimedia Commons, CC BY 2.0) → nine contact sheets, up to nine photos each (triage reads, relative context) → 21 dedicated reads (1536 px, one photo each) → 21 face crops (enrichment) → 16 repeat reads (8 groups × 2, identical settings).

model gemini-3-flash-preview
run envelope 67 provider calls, 135 judgment records, 34m23s wall clock.
Diagram of the four audit passes from 77 photos to repeat reads

Scroll horizontally to see the full diagram ↗

The set, and why it is this one

The demo runs on one coherent, openly licensed event set — fixed and licensed before any model touched a frame.

  1. Scale: 60–120 photos — large enough that frames compete for the same slot, small enough to inspect every outcome by hand. This set uses 77.
  2. Coherence: one event, not a stock collage. Flips are meaningful when near-duplicate frames from the same session are judged against each other.
  3. License clarity: only per-asset CC0, CC BY, or CC BY-SA with a stable author and source record. Anything unclear is excluded.
  4. Fixed in advance: the set was licensed, SHA-256 verified, and frozen before the run — selection followed the event, not the results.
  5. Manifest-first: every asset carries a source URL, creator, license, attribution text, and a content hash. The manifest is the authority.
  6. No redistribution: the repository ships the manifest and a fetch script; photographs stay at their source, and this page hotlinks them with attribution.
photos
77 — one event: London 2012 opening ceremony
creator
Nick Webb (London, United Kingdom)
license
CC BY 2.0 — verified per asset
integrity
SHA-256 per file; manifest in examples/open-demo
acquisition
scripted, fetch-only — no image files committed

Every number on this page reproduces from the frozen artifacts in the repository: examples/open-demo/run-2026-10-03 (audit.json, report.md, RUN-NOTES.md).

03 / disagreement

Six finalists fell on a dedicated read.

The triage pass judges a photo inside a contact sheet; the dedicated pass reads each finalist alone, at higher resolution. Of the 21 paired finalists (all triage accepts), 6 flipped — every one downward: 2 accept→reject, 4 accept→maybe; 15 stayed accept. The other 56 photos had no dedicated comparison. This run shows the disagreement; it does not isolate whether sheet context, resolution, or other stage differences caused any single change.

15accept → accept
4accept → maybe
2accept → reject

These six cards are every flip among the 21 paired photos — the complete set of verdict changes, nothing curated out.

What the triage pass saw

Relative reads happen inside sheets like this one: nine frames judged together in a single call — sheet 7 of 9. Two of this sheet's frames flipped downward when read alone, marked below; whether the neighbours caused the change is not something this run isolates.

1 Danny Boyle — Nick Webb, CC BY 2.0, Wikimedia Commons 2 Green and Pleasant Land — Nick Webb, CC BY 2.0, Wikimedia Commons 3 Here's To Everyone Who Gives Their Best — Nick Webb, CC BY 2.0, Wikimedia Commons 4 Mike Oldfield — Nick Webb, CC BY 2.0, Wikimedia Commons 92 → maybe 555 Olympic Rings Converge — Nick Webb, CC BY 2.0, Wikimedia Commons 6 Olympic Stadium — Nick Webb, CC BY 2.0, Wikimedia Commons 7 Orbit & The Olympic Stadium — Nick Webb, CC BY 2.0, Wikimedia Commons 8 Royal Box — Nick Webb, CC BY 2.0, Wikimedia Commons 9 Smoking — Nick Webb, CC BY 2.0, Wikimedia Commons 86 → reject 35

Tiles are hotlinked Wikimedia Commons thumbnails; each links to its source page. Photographs: Nick Webb, CC BY 2.0.

The six flips, with the actual photographs — scores are composites (0–100):

04 / repeatability

Ranking is not stability.

The top eight finalists were read twice under identical settings. All eight came back robust: identical verdicts, composite spread within the v1 threshold (mostly 0; one spread of 3). A verdict that survives a re-read is evidence; a rank position is not.

two reads per selected photo · identical verdicts · composite spread ≤ 5 under v1

0002 robust spread 0
0008 robust spread 0
0015 robust spread 0
0016 robust spread 0
0019 robust spread 3
0022 robust spread 0
0033 robust spread 0
0072 robust spread 0

This tests only these two configured repeats for these eight photos — not ranking stability, and not general model reliability.

05 / accounting

This run's estimated usage cost.

Estimated across 67 calls: $0.273687 from a pinned price table (documented in the repository). 67 records carry usage for 67 calls; the other 68 records — triage siblings sharing one billed call per sheet — have no separate bill. "Known $0" means no known-price records, not free provider usage.

triage $0.0775845 9 calls
dedicated $0.102256 21 calls
face $0.0537015 21 calls
repeat $0.040145 16 calls
estimated total $0.273687
token envelope

80,928 in · 18,559 out · 59,182 thinking

thinking-token counts may overlap output-token counts (per the frozen report's warnings)

Known / estimated / unknown stay separate. No guessed bills.

06 / method

Run it, inspect it.

Try the audit mechanics — keyless and image-free, on invented records:

git clone https://github.com/rmax-ai/cull-audit
cd cull-audit
uv venv && uv pip install -e .
python -m cull_audit demo --output tmp/demo

This synthetic demo exercises the mechanics; it does not reproduce the 77-photo study.

Check the study — rerun the audit on the frozen judgments:

python -m cull_audit audit \
  --judgments examples/open-demo/run-2026-10-03/judgments.json \
  --baseline-stage triage --decisive-stage dedicated \
  --prices examples/open-demo/price-table-gemini-3-flash-preview-2026-10.json \
  --output tmp/audit-check --source-date-epoch 1791029022

Produces the same aggregates as the committed audit (6/21 flips, 8/8 robust, $0.273687). Byte-identical output means rerunning the audit on identical frozen inputs — not rerunning the live provider; the staged-workspace procedure is in RUN-NOTES. The optional reference runner (your own key, dry-run gate first) is how the study itself was produced.

Frozen run artifacts ↗README ↗Judgment schema ↗

07 / boundary

What this cannot say.

One licensed set, one model snapshot, one prompt configuration: these are demonstration receipts, not a benchmark. No claim of human ground truth; provider outputs drift; aesthetic judgment stays with people.

The audit is the point, and it is open to challenge: inspect the frozen audit.json and report.md, run the synthetic smoke test, or bring your own records and pressure-test the pairing rules, stability thresholds, and cost attribution.

Import guide ↗Contribution guide ↗
Back to the protocol ↑
return to top ↑