Accuracy
How often each part of the data matches its printed page, how we found out, and what we haven't measured yet. Every figure asks one question: does the page say this? A record can match its page and the page can still be wrong. We check the record, not the world.
The audit figures were measured on release v0.6; v0.7.1 has not been re-measured yet. No person has checked a random sample yet. These figures come from a model reading the page, or from keys the records write themselves. Read them as good estimates, not ground truth.
In short
| Field | How often it matches the page | Sample |
|---|---|---|
| One person's vote | 342 of 349 (98%) | v0.6 audit, page text |
| Amount of money | 333 of 357; 22 not on the linked page, 2 at the wrong scale | v0.6 audit, page text |
| Date, when there is one | 117 of 123 (95%) | v0.6 audit, page text |
| Outcome (passed, referred, …) | 254 of 298 (85%); 27 of the 44 misses are first readings shown as passed | v0.6 audit, page text |
| Kind of item | 268 of 320 (84%) | v0.6 audit, page text |
| Roll-call totals | 173 of 199 (87%) | v0.6 audit, page text |
| Roll-call names complete | 163 of 200 (82%), worst before 1930 | v0.6 audit, page text |
| Ordinance or resolution number, older passes, 1906–1997 | 251 of 360 (70%) | v0.6 audit, page text |
| One-line summary without a real problem | 315 of 359 (88%) | v0.6 audit, page text |
| Place, when the minutes give an address | right block 92.8% of 2,883 | the record's own key |
| Place, at a corner | right block 51.9% of 291 | the record's own key |
| Page link | 620,576 of 539,192 decision rows matched to the viewer's page number (v0.7.1) | page_exact |
How we checked
A model audits the page text (16 October 2026, on v0.6). For each kind of record we drew 200 to 360 at random, spread across decades. Model checkers read the OCR text of the cited page, judged each field, and quoted the words that show any error.
The record's own key (6 October 2026). Some decisions print their own assessor block. We hide that line, place the decision from the rest of its text, and see whether we land on the printed block: 3,195 checks, no model in the loop.
A model re-reads the page image (15 October 2026, on v0.5 facts). 300 facts at random; Claude Opus 5.5 looked at the scan each fact links to. Votes 48 of 50, money 43 of 45, places 40 of 45, outcomes 37 of 45.
Agreement with an answer key. For 40 San Francisco pages (115 decisions), a stronger model wrote out every decision in full, and we score the main reading against it: decision found 0.95, number 1.00, vote counts 0.99, vote names 0.95, money F1 0.80, places F1 0.66. No person checked that key, so this is agreement between two models.
Place links by precision
| How the decision was placed | Seriously off in the v0.6 sample |
|---|---|
| Address | 5 of 48 |
| Parcel | 3 of 20 |
| Block | 0 of 25 |
| Stretch of street | 22 of 49 |
| Corner | 28 of 46 |
| Whole street | 53 of 68 |
"Seriously off" means the wrong place, a place the text doesn't name, a street named only as a boundary ("between X and Y"), or a link much vaguer than the text allows.
Not measured yet
- a random sample checked by a person (the sheets are drawn: 100 decisions, 300 vote names);
- whether each printed name is matched to the right supervisor (99.4% of names are matched to someone);
- the topic labels on older decision rows.
The measured scores, with sample sizes, are in the field_scores table.