Methodology
How the release is built, step by step, and what it holds.
scripts/night_build.sh v0.4 runs everything, one step at a time (DuckDB capped at 2 GB and 2 threads), checks, then publishes:
| step | script | what |
|---|---|---|
| 1 | 01_supervisors_roster.py | the supervisor roster (names, years served, Wikidata ids) from Wikipedia's list |
| 2 | 20_rollcalls.py | every printed roll call in the Board text -> roll calls + one vote per supervisor |
| 3 | 10_stage.py, 11_stage_extra.py, 25_v2.py | the map's files, the extraction runs, decisions v2, the media registry, the appraisals and the DL tags -> claim-shaped rows |
| 4 | 80_showcase_fetch.py, 81_showcase_stage.py | the SF proof: metadata search across the Archive, Open Library and the Wayback Machine |
| 5 | 15_ia_meta.py | archive.org metadata of every cited item; Democracy's Library membership by collection |
| 6 | 30_build.py | the DuckDB build (spec v1.0 tables, vocabularies, claim ids, support, redirects) -> Parquet |
| 7 | 35_geometry.py | the place graph: geometries, relations, places_at() |
| 8 | spec/tests/run_checks.py | the spec's checks (stops on an error) |
| 9 | 40_export.py, 41_public_scrub.py | the public, privacy-filtered SQLite and Parquet (the full tables stay local); every free-text column of embargoed rows filtered, and the build stops if a name pattern is left |
| 10 | 45_docs.py, 46_pages.py, 50_entities.py, 70_products.py | dictionary, release pages, one JSON per entity, the map's block files regenerated from claims and diffed |
| 11 | 60_publish.sh | upload to R2 |
Inputs (scripts/config.json) are local today: the Board corpus, the re-sweep and decisions-v2 run folders, the map's files and the leaf maps. Publishing them as a qipu-data-inputs archive.org item is on the list (review 1, M7).
What the release holds
Counted from the published tables of v0.7.1:
| Count | |
|---|---|
| decisions (each printed item once) | 539,192 |
| decision rows (some items were read more than once) | 718,093 |
| rows from the second, closer reading | 344,936 |
| claims | 8,367,105 |
| share of claims citing an archive.org item | 86.3% |
| vote claims | 4,724,389 |
| printed roll calls | 171,123 |
| persons (officials) | 294 |
| places with stable ids | 470,596 |
| claims at a door (address, parcel or building) | 1,543,612 |
| sources | 6,377 |
| cited archive.org pages | 183,888 |
Use
- Explorer: https://tiles.qipu.org/data/ (questions, the SF proof, click anywhere, SQL over SQLite or over the Parquet).
- Parquet:
releases/v0.7.1/parquet/<table>.parquet(the public, privacy-filtered tables). - Datasette:
pip install datasette && datasette qipu_sf.db -m metadata.json. - AI (MCP):
mcp/qipu_data_mcp.py, six tools:place_history(address),person_votes(name, years),search_decisions(text, years),places_at(lat, lon, date),showcase(place), andverify_claim(claim_id)(a model re-reads the claim's scan page; rate-limited and spend-capped; it checks the record, not the world). Every answer carries its archive.org links; cite them as "SF Board of Supervisors, Journal of Proceedings, <date>, archive.org/details/<id>/page/n<n>". Run:QIPU_DB=/path/to/qipu_sf.db python mcp/qipu_data_mcp.py, orclaude mcp add qipu-data -- python /path/to/mcp/qipu_data_mcp.py. - Benchmark:
bench/(golds, prompt, scorer) to test a model on public records.
Known limits
- Counts: decision rows come from several extraction passes, and one printed item can have up to five rows. Count distinct decisions with
decision_group(one id per printed item); its duplicate matching is about 0.92 precise and 0.70 complete. - Roll calls are re-read from the page text since v0.6.1, but some across page breaks are still incomplete (1959 hand check: 98 of about 102 roll calls with a No are found).
- Street links: on the audit sample, 39 of 68 street links and 20 of 46 corner links are still seriously wrong.
- About one numbered decision in five before 1998 carries the number of a neighbouring item.
- About 8% of decisions v2 rows have no date (no dated heading on their page).
- Many claims are citywide (votes); the table above gives the door-level count. Votes sit at the city, not at what was voted on.
- Feature geometry (creek, shipyard, freeway, landfill, redevelopment area) is a circle around the Wikidata point, marked approximate.
- The accuracy numbers are agreement with Opus-written keys, not hand-checked samples (except the appraisals); no held-out set, no intervals yet.
- Vote confidence is 0.87 (the measured roll-call count rate on the SF key, n = 46), 0.6 where the printed counts disagree with the names; person matching is unmeasured (a 300-name hand check is ready: build/handcheck/vote_names_300_HOWTO.txt).
- Date ranges: until v0.6.2, 1.68 M vote claims of the v0.6.1 re-read had
t_frombut not_lo/t_hi; v0.6.2 fills them and check T6 stops a build that misses any. - The name hider is a pattern (two or three capitalised words), not a full name recogniser: it over-hides some places and can miss a lone surname. Raw quotes of embargoed records are not published at all.
- OpenSFHistory photos are link-only in our products until WNP agrees to inline display.