qipu data›San Francisco›Documentation›Accuracy

Accuracy

How each field was made and how well it is known. "Unmeasured" means no gold exists yet; nothing here is a guess.

field	made by	reliability
x_roll_calls: counts (ayes/noes/absent)	rules (20_rollcalls.py)	Rule check on all 171,123 roll calls: printed counts equal the names parsed in 90.5% (1,402 print no counts). SF gold (decisions_bench/gold.jsonl, written by Opus 5.5, no record of a hand check): counts exact 40/46 (87%) where the roll call is in the item text; 12 of 58 gold roll calls not found (mostly outside the gold text span).
x_roll_call_votes.person_id (who voted)	rules: surname + years served, official roster	99.4% of 2,084,507 printed names matched to a person (659 ambiguous, left empty). Gold names: 536/541 matched. Correctness of each match: unmeasured (no gold of names to persons yet); confidence in `claims` drops to 0.6 when the counts do not match.
claims voted: value_text (yes/no/absent)	rules	Same as the roll-call counts above; unmeasured separately.
decisions.block, concerns_place (block level)	rules (map block index)	Measured 2026-10-06 on 3,195 checks (accuracy.json): 88.9% on the exact block, 91.6% within one block face, 95.5% within 250 m. By tier: address 92.8%, corner 51.9%, segment 71.4%. Street-tier placements (a street, not a block) are not scored.
pages.display_n / page links	per-volume leaf map: the audited Journal map, IA scandata for the other items (qipu_data.leafmap_ia)	179,043 of 180,775 pages are mapped OCR page -> viewer page (`page_exact`); the rest keep n = OCR page and may be a page or two off. iiif_url is $<display_n + 1> (that IIIF service counts the shown pages from 1). Gate L1 checks the numbering; L2 checks that a fixed sample of quotes is on the cited page.
decisions.topic / subtopic / action (bos_corpus)	rules + title classification (build_core_table.py)	unmeasured
decisions from the re-sweep: outcome, topic, plain, place (claude-haiku-5-5, RESWEEP_PROMPT.md)	LLM, one item at a time	unmeasured for this prompt. Indicative only (same model, page-level prompt, SF gold, L4_TEST.md): action 0.97, topics F1 0.84, places F1 0.89.
decisions.plain_best (the map's one-line summaries)	LLM + accuracy pass (CHECK_PROMPT.md)	unmeasured as a rate; lines the accuracy pass found wrong were rewritten or dropped.
instruments (numbers)	regex on the item text	unmeasured. OCR can garble numbers (1G2 for 162); those are not parsed.
address_resolution	exact text match to DataSF EAS	exact matches only, by construction; unmatched texts are not resolved.
sources.in_democracys_library	rule: the item's collections vs DL scope (DL_INVENTORY.md)	exact by definition, as of the metadata fetched 14/10/2026.
built (assessor year built), fire_1906, sanborn lines	official records (DataSF, Sanborn via OldInsuranceMaps)	record values, not extracted; the Sanborn line wording is a convention of the map (unmeasured).
dl_documents links (archive items to places)	IA full-text search + relevance score	unmeasured; `confidence` holds the search relevance score.
x_dl_tags (places a DL item is about)	LLM on metadata + first OCR	unmeasured (no gold of document places yet).
media rights_note	rules by holder, on top of the licence the map recorded	rule output; the item page is the authority. 17 items have no holder rule (rights not recorded: link only).
x_decisions v2 money currency (SF)	fixed in the build	the run's writer stamps CAD on every amount; SF amounts are relabelled USD in the release (run files unchanged); gate M1 fails any SF amount in CAD.
stories, timelines, doors, dateline	written for the map (LLM + hand review on demo streets)	unmeasured; each line cites its source.
x_decisions run sf-v21-haiku-1014 / van-v21-haiku-1014 (decisions v2)	LLM, one call per page	agreement with an Opus-written key (L4_TEST.md, SF gold; not hand-checked; n = 115, a Wilson 95% interval on 0.95 is about 0.89-0.98): finds 0.95, number 1.00, action 0.95, vote counts 0.99, all three 0.95, vote names 0.95, mover 0.93, money F1 0.80, places F1 0.66, parties F1 0.61, dates F1 0.55, related F1 0.17 (new fields scored against an Opus key, not hand-checked).
x_appraisals (SFRA property files), claude-haiku-5-5	LLM, one call per page	measured on 18 pages (ia_records/sfra/bench/score.json): address 18/18, parcel 18/18, owner 18/18, present_use 17/18, rent 16/18, appraised_value 18/18.

Measured field scores

run	field	metric	value	gold
andrews-sonnet55-2026-10	andrews owners	confirmed on the linked page image / checked (Wilson 95% 0.83-0.99); re-checked on v0.6 after the page-link fix (v0.5 links were one page off)	0.97	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
bos-corpus-2026-09	outcomes	confirmed on the linked page image / checked (Wilson 95% 0.25-0.84)	0.57	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
bos-corpus-2026-09	places	confirmed on the linked page image / checked (Wilson 95% 0.70-0.96)	0.88	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
dl-sf-facts-v1.1-haiku55-2026-10	dl facts	confirmed on the linked page image / checked (Wilson 95% 0.79-0.96); re-checked on v0.6 after the page-link fix (v0.5 links were one page off)	0.91	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
rollcall-parser-v1	counts	exact where the roll is in the text	0.87	decisions_bench/gold.jsonl roll calls (Opus-written key), scripts/22_rollcall_gold.py
rollcall-parser-v1	names to persons	matched	0.99	decisions_bench/gold.jsonl roll calls (Opus-written key), scripts/22_rollcall_gold.py
rollcall-parser-v1	votes	confirmed on the linked page image / checked (Wilson 95% 0.70-0.99)	0.93	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
sf-v21-haiku-1014	action	exact	0.95	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	dates	F1 (vs Opus key)	0.55	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	finds	recall	0.95	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	interest	Spearman	0.78	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	money	F1 (vs Opus key)	0.80	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	money	confirmed on the linked page image / checked (Wilson 95% 0.85-0.99)	0.96	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
sf-v21-haiku-1014	mover	exact	0.93	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	number	exact	1.00	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	number+action+votes	all exact	0.95	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	outcomes	confirmed on the linked page image / checked (Wilson 95% 0.73-0.94)	0.87	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
sf-v21-haiku-1014	parties	F1 (vs Opus key)	0.61	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	places	confirmed on the linked page image / checked (Wilson 95% 0.70-0.97)	0.90	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
sf-v21-haiku-1014	places	F1 (vs Opus key)	0.66	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	related	F1 (vs Opus key)	0.17	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	topics	F1	0.90	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	vote names	exact	0.95	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sf-v21-haiku-1014	votes	confirmed on the linked page image / checked (Wilson 95% 0.85-0.99)	0.97	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
sf-v21-haiku-1014	votes (counts)	exact	0.99	decisions_bench/gold.jsonl (SF, 40 pages: an Opus-5.5-written key, no record of a hand check) + v2 Opus key
sfra-appraisals-haiku55-2026-10	address	normalized exact (18 pages)	1.00	ia_records/sfra/bench (18 pages)
sfra-appraisals-haiku55-2026-10	appraised_value	normalized exact (18 pages)	1.00	ia_records/sfra/bench (18 pages)
sfra-appraisals-haiku55-2026-10	owner	normalized exact (18 pages)	1.00	ia_records/sfra/bench (18 pages)
sfra-appraisals-haiku55-2026-10	parcel	normalized exact (18 pages)	1.00	ia_records/sfra/bench (18 pages)
sfra-appraisals-haiku55-2026-10	present_use	normalized exact (18 pages)	0.94	ia_records/sfra/bench (18 pages)
sfra-appraisals-haiku55-2026-10	rent	normalized exact (18 pages)	0.89	ia_records/sfra/bench (18 pages)
sfra-appraisals-haiku55-2026-10	sfra appraisals	confirmed on the linked page image / checked (Wilson 95% 0.74-0.95), after 6 checker-prompt errors were corrected (see ACCURACY_v0.6.md)	0.89	eval/model_scan_check (300 public claims re-read against the page image by claude-opus-5-5; no person checked them)
van-v21-haiku-1014	money	F1 (vs Opus key)	0.73	decisions_bench Vancouver gold (30 pages, 48 decisions: Opus-written, 21 decisions hand-checked)
van-v21-haiku-1014	mover	exact	1.00	decisions_bench Vancouver gold (30 pages, 48 decisions: Opus-written, 21 decisions hand-checked)
van-v21-haiku-1014	number+action+votes	all exact	0.98	decisions_bench Vancouver gold (30 pages, 48 decisions: Opus-written, 21 decisions hand-checked)
van-v21-haiku-1014	vote names	exact	0.98	decisions_bench Vancouver gold (30 pages, 48 decisions: Opus-written, 21 decisions hand-checked)