Case study F
Assembling a photo library for hundreds of named minors, using automated search, in a field where dozens of them share a name. Every design decision came from one sentence I wrote early and kept: being wrong in one direction costs far more than being wrong in the other.
Publishing about someone reads far better with a photograph of them. Gathering hundreds of photographs by hand is impossible on a daily cycle, so the search had to be automated. The subjects are named minors, and in this field shared names are not an edge case — they are routine.
So the failure mode isn't “no photo found”. It is finding a photograph of the wrong child and captioning it with this child's name. Both children are real. Both images are correctly labelled at their own source. The mistake happens entirely on my side, and it is not the sort of mistake you can quietly correct later.
A name is not an identifier. It is a search term that usually works.
There was a second, quieter problem. When a report says a subject has no photograph on file, that could mean no photograph exists — or that the matcher is broken and every cell is empty. Those two produce identical output, and only one of them is a reason to relax.
And four limits that were never up for negotiation: no likeness of a named minor is ever generated; third-party images of minors are not collected without an explicit per-source decision; the consented internal library is always checked first; and a name is never matched to a social media account by guesswork — a request I declined three separate times, offering slower routes instead.
One file per subject folder, one row per image: the search string used, the page the image sat on, the image's own address, that page's title, a timestamp — and the field that does the work, the evidence that this is the right person.
| Verdict | Condition | Action |
|---|---|---|
| Pass | The source page's own title names this subject's affiliation, or their region, and nothing contradicts it | Usable |
| Wrong | The title names a different affiliation. Same name, different person | Discarded. The dangerous class |
| Weak | The title names neither. It may well be them; nothing in the record says so | Held for a human |
A real case from the archive: a search for one subject returned a portrait of someone with the identical name at a different institution in another state — a real person, correctly captioned at her own source, and entirely the wrong person for us. Nothing but the affiliation check would have caught it.
A source title naming an affiliation that is in our set but is not this subject's also counts as Wrong. That is precisely how a teammate's or an opponent's photograph gets filed under someone else's name — the match looks locally plausible because everything in it is familiar.
Pass three is the only inference in the system, and it is labelled, logged and reviewed. Compare this with an earlier project of mine where a hand-written short-name dictionary did the same job silently and failed invisibly — this is that lesson, applied.
Both stores contain subject folders that exist and hold nothing. An empty folder counts as No, and is reported separately from never having been collected — because “we started and nothing landed” and “we never started” need different responses from whoever picks it up.
2,285 images across 234 subject folders. A per-subject audit that answers the only question anyone actually asks — is there a usable image, yes or no — with the supporting detail beside it, so a yes can be checked and a no can be chased. Uncertain matches reach a person instead of being resolved by the machine.
And the number I'd rather state than have someone else find: 1,655 of those 2,285 images sit in folders carrying a provenance file. The other 630 do not. That is 72% coverage, across 133 of 209 image-bearing folders. Twenty-six further folders are empty.
A case study about provenance that didn't measure its own provenance coverage would be making exactly the mistake it's about.
The 28% gap isn't carelessness in the moment — those images were collected during earlier runs, before the provenance schema existed, and were never backfilled. But the effect is the same as if they had been: 630 images whose origin nobody can now verify, sitting in the same library as 1,655 that can be, looking identical.
The fix isn't discipline, it's a gate. An image without a provenance record should not be able to enter the library at all — and anything already inside that lacks one should be quarantined rather than silently trusted. I wrote a rule where I needed a check, which is the third time that exact habit has appeared across these case studies.
A second collection run introduced a new provenance format instead of writing the existing one, so the archive now holds two file types recording the same thing in different shapes. Every consumer has to understand both, and any future one will quietly support whichever it met first.
One schema, versioned, with a migration for the older files. It was a twenty-minute decision I skipped because the new run was in a hurry.
Pass, Wrong and Weak are derived at audit time from the stored page title. That keeps the raw evidence honest, which is right — but it means a verdict can change silently if I refine the rule, and there is no record that a human already reviewed and cleared a Weak case.
Both should be stored: the evidence and the verdict, stamped with the rule version that produced it and with any human decision recorded against it. Otherwise every review is done again, or worse, silently undone.
The positive control. It cost almost nothing — run the matcher against cases whose answer I already know, before believing anything it says about cases I don't. Every pipeline that reports absences should have one, and almost none do, because a confident empty result looks exactly like a correct one.