The working record

Backs the claim: the spec comes before the work

This is the section structure of a real build specification I wrote before starting a dataset. Nothing here is generated after the fact — this document existed before a single row did, and the build was run against it.

## What I want built
### Deliverable per item

## Scope — what I already know about the landscape
    ### [authority 1] — about 104 sponsoring institutions
    ### [authority 2] — recognised since 2021
    ### [authority 3]
    ### [authority 4]
    ### A cross-source shortcut worth using early
    ### Sources to start from

## Rules — these are not negotiable, they were learned the hard way

## Traps that cost real time on the previous build

## What already exists in this repo — reuse it, do not rebuild it
    - extractor      renders JS, returns page text
    - sync layer     sheet I/O and routing
    - change watcher  currently filtered; widening it is one line
    - registry lookup city/state/website, no key needed
    - overview doc    renames and moves from the last build
    - contract doc    the extraction contract. Read it first.

## How to work

Real document, 11 KB. Authority names and filenames replaced with [bracketed] descriptions. Two headings do the real work here: “Rules — not negotiable, learned the hard way” and “Traps that cost real time on the previous build”. Both exist because I wrote down what went wrong last time instead of rediscovering it.

Backs the claim: resumable batch jobs · idempotence

My most-repeated rule, in the code that taught it to me. The batch pointer is a single integer on disk. It only moves once the write has actually landed.

def load_state():
    try:
        with open(STATE_FILE, encoding="utf-8") as f:
            return json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        return {"last_processed_row": 0}

def save_state(state):
    with open(STATE_FILE, "w", encoding="utf-8") as f:
        json.dump(state, f)

try:
    # Write all batches at once (avoids rate limit)
    if review_rows:
        review_ws.append_rows(review_rows, value_input_option="USER_ENTERED")
    if queue_a_rows:
        queue_a_ws.append_rows(queue_a_rows, value_input_option="USER_ENTERED")
    if queue_b_rows:
        queue_b_ws.append_rows(queue_b_rows, value_input_option="USER_ENTERED")

    # Save state only after successful write
    if not args.all:
        save_state({"last_processed_row": 0 if end >= total else end})
    print(f"✓ Wrote {len(review_rows)} review rows.")

except Exception as e:
    print(f"✗ Sheet write failed: {e}")
    print(f"  Data NOT saved to Sheet. state.json NOT updated.")
    sys.exit(1)

Verbatim, except two queue variables renamed. The comment “Save state only after successful write” is mine and is in the original. The failure path is the point: it says out loud that the pointer did not move, so a re-run redoes the batch rather than skipping it.

Backs the claim: auditing structure, not just contents

This is the script from case study A — the one that found 157 duplicate clusters and thousands of orphaned links in a dataset whose contents I had verified twice from primary sources. It loads all four tables and simply asks which rows point at parents that do not exist.

"""
Comprehensive data audit. Reads the workbook downloaded from the
live sheet and emits findings as JSON, which the report generator
turns into audit_report.md.
"""

entities  = load("Entities")
offerings = load("Offerings")
people    = load("People")
roles     = load("Roles")

F["counts"] = {
    "entities":  len(entities),
    "offerings": len(offerings),
    "people":    len(people),
    "roles":     len(roles),
}

# a role whose offering_id resolves to nothing is an orphan
offering_label  = dict(zip(offerings["offering_id"], offerings["_label"]))
roles["_label"] = roles["offering_id"].map(offering_label)
orphans         = roles[roles["_label"].isna()]

F["roles_orphan_offering_count"] = len(orphans)

Verbatim, with the four table names replaced by neutral equivalents. Six lines of .map() and .isna() is all it took — which is exactly why I run it on a schedule now instead of waiting until something feels wrong.

Backs the claim: provenance captured at collection, 1,655 times

One CSV sits beside every subject's assets. This is the real header row and a real record, with the two identifying columns replaced.

"File","Subject","Affiliation","Search","SourcePage",
 "ImageUrl","ResultTitle","MatchEvidence","RetrievedAt"

"asset-01.jpg", "[subject]", "[affiliation]",
"\"[subject]\" [descriptor] [region]",
"https://[video-platform]/watch?v=…",
"https://[cdn]/…/maxresdefault.jpg",
"[subject] Highlights (Class of 2024) - [platform]",
"name in image result metadata",
"2026-09-09T08:17:49.175-06:00"

Real schema, real timestamp precision, real structure. The column that matters is MatchEvidence — it records why I believed this asset belonged to this person. Where that column can't be filled honestly, the asset is withheld rather than used. 149 of these files exist across 234 subject folders.

Backs the claim: scale of what was actually produced

A count of what is sitting on my drive right now. Not estimates — these are file counts.

The two numbers I'd point at: 22 written specifications, because it shows the specs weren't a one-off; and 76 delivery folders, because a pipeline that ran once proves nothing.

Backs the claim: systems thinking across two conflicting constraints

A legitimate technical term in this field is also a word that automated moderation on social platforms reads badly — and we publish about named minors, so the risk wasn't worth it. But that term is exactly what people type into a search engine, so removing it from the writing would have cost real traffic.

Two constraints, apparently opposed. The resolution is a single observation: a search engine cannot read text baked into a PNG.

// Every field here overrides the top-level field of the
// same name, for EVERY template. Absent block = render unchanged.
"wording": {
  "slug":      // social exports only — the blog master and OG crop
                 // keep the real slug, because a filename on the site
                 // is a search signal and one on social is invisible
  "headline":  // cover + blog headline
  "statLabel": // what the big number is called
  "eyebrow", "subtitle", "coverBullets", "coverBlurb",
  "groupSlideTitle", "notes"
}

Real structure, real comments, field list intact. The artwork gets platform-safe wording at zero cost to the page's ranking, while the prose keeps the term people actually search for. Alt text in the manifest describes what the graphic really says, so the keyword still survives in the one place a crawler reads.

Backs the claim: analysis written to be acted on

The audit from case study A didn't stop at a script. It produced a report with every finding ranked by severity and scaled against the table it affects — so whoever picks it up knows what to fix first and how big the job is. This is its executive summary, redacted.

Real report, real counts. Column headings and finding text are mine. Two things I'd point at: severity is split into governance, integrity and consistency rather than one blurred scale, and every finding carries “n of total” — so nobody has to guess whether 877 is a crisis or a rounding error. Exact row references follow in the full document.

Backs the claim: pipeline architecture · idempotent batch processing

Some sources render as plain pages. Others only exist once a browser has run their JavaScript. One fetcher cannot do both, so there are two — and everything downstream of them is deliberately shared, because two pipelines that drift apart are worse than one pipeline that can't reach every source.

Drawn from the running code, not from memory. The shared-contract box is the part that has to be defended — when the two fetchers were changed independently, that box is where the drift would have shown up first, which is why the extraction schema and the prompt live in one file and are imported by both.

Backs the claim: machine-readable delivery · honest handover

Every rendered package ships with a manifest. It records what was produced, at what size, with what alt text — and, the part I care about most, it records what is missing, by name, so nobody downstream has to discover the gap themselves.

{
  "article":    "[title] ([month year])",
  "articleId":  "2026-08-[slug]",
  "renderedAt": "2026-08-30T03:46:59.492Z",

  "outputs": [
    { "file": "…-blog-1920x1080.png",
      "width": 1920, "height": 1080,
      "alt": "[title] — [brand]" },

    { "file": "…-social-01-cover.png",
      "width": 1080, "height": 1350,
      "alt": "[title] — carousel cover" },

    { "file": "…-social-02-[subject].png",
      "width": 1080, "height": 1350,
      "alt": "[subject], [role], [affiliation] — No. 1 with
              541 [metric], [title]" }
    // … one entry per slide
  ],

  "photosMissing": [
    { "rank": 11, "name": "[withheld]", "school": "[withheld]", "state": "TX" },
    { "rank": 13, "name": "[withheld]", "school": "[withheld]", "state": "GA" }
    // … 20 entries in this delivery
  ]
}

Real keys, real timestamp, real sizes. Names and affiliations replaced with [withheld]. Two things worth noticing: the alt text is written to be genuinely descriptive — rank, figure and role, not “image of a person” — which is the one place a crawler can read what a PNG says. And photosMissing is a first-class field, not a note in an email. If twenty subjects had no usable photograph, the delivery says so in a form a script can read.

Backs the claim: handover under conflicting instructions

Each batch ships with a written note. It opens with status — everything is a draft, nothing is live, only two named people publish — then the deliverables and their counts. Then this, which is the section I'd want anyone to read:

## One instruction in the daily brief was not followed, deliberately

The brief says: "[term] must NOT appear on any graphic
or in any caption. Use [alternatives]."

The client reversed that on 28–29 August — "[term] is the proper
term" — and the first six were re-rendered to match. These
six follow the client. Every graphic and caption uses it.

The original reason for the ban was platform moderation on
posts about named minors, and that risk has not gone away;
it is just a call the client has made and owns. If the brief's
rule is meant to stand again, say so and the graphics re-render
in about ten minutes — the mechanism for making artwork and
prose differ is still in the code, unused.

Real section, with the term and the names removed. Two stakeholders gave opposite instructions. I followed one, said clearly which and why, restated the risk that hadn't gone away, named who owns the decision, and quoted the cost of reversing it. The note ends with a corrections log — things I changed in the draft and the arithmetic behind each one.

I could show more, and it would be more persuasive. I've chosen not to, and I'd rather say what I'm holding back than let this page look complete.

• Screenshots of the live systems Every screen I work in is full of real names and real contact details. There is no crop that fixes that, and a heavily redacted screenshot proves less than the code that produced it.
• The datasets themselves They're the commercial asset of the business I work for, and they contain thousands of real people's contact details. Structure and counts, yes. Rows, never.
• Anything identifying a subject Some of the people in this data are minors. Their names, images and handles don't appear here in any form, redacted or otherwise.
• Available on request If you need more than this, I keep a fuller evidence pack — real screens, real outputs, whole scripts — that I can walk through directly in a conversation. Just ask.