What a record holds
The release is a set of tables (Parquet, SQLite, CSV) plus one JSON document per place and person. Everything carries a stable id that starts with qipu:.
The main tables
| Table | One row per | Rows in v0.7.1 |
|---|---|---|
decisions | item the Board acted on, per reading | 718,093 |
claims | single statement, with its source page | 8,367,105 |
x_roll_calls | printed roll call | 171,123 |
x_roll_call_votes | name in a roll call | 2,084,507 |
instruments | ordinance, resolution or bill, by printed number | 72,224 |
places | block, parcel, address, street or named place | 470,596 |
persons | supervisor or mayor | 294 |
pages | cited archive.org page | 183,888 |
claim_reliability | claim, with its reliability label | 8,367,105 |
The data dictionary defines every column of all 45 tables.
A decision row
The columns you'll use most:
| column | what it holds |
|---|---|
decision_group | the decision a row belongs to: rows of the same printed item share it (539,192 decisions) |
decision_id | the stable id, e.g. qipu:sf/decision/journalofproceed1912sanfrich:66/2 (volume, page, item) |
year, date | when; date is empty when the page has no dated heading (31,492 of 539,192 rows in v0.7.1); date_from says where the date was read |
kind | ordinance, resolution, motion, claim, hearing, appointment, … |
outcome_best | passed, refused, referred, … (best available reading) |
plain_best | one-line summary, written by a model |
quote | the printed lines it was read from |
source_url | the archive.org page |
page_exact | true when the link was matched to the viewer's own page number |
summary_by | which model wrote the summary |
A claim
A claim is one statement with its source: "this decision concerns Islais Creek", "Supervisor Murdock voted yes". Each claim has a subject_id, a property, a value (value_text or value_num), its quote, its source_id and page_url, and a reliability label (tier in claim_reliability, joined on claim_id):
- verified (889 claims): a person checked it, or two independent documents agree;
- quoted (2,962,695): the line it was read from is on the cited page, and the reading method measures 0.9 or better;
- unverified (5,403,521): everything else.
Ids
Ids are stable across releases. When a record is merged or retired, its old id keeps working and points to the new one (135,661 redirects in v0.7.1). The shapes:
qipu:sf/decision/<volume>:<page>/<item> qipu:sf/place/block/0847 qipu:sf/place/address/eas-289463 qipu:sf/place/creek/islais-creek qipu:sf/person/dianne-feinstein
JSON documents
Every place and person has a JSON document with all its claims and their sources:
curl https://tiles.qipu.org/data/sf/place/creek/islais-creek
{
"id": "qipu:sf/place/creek/islais-creek",
"type": "place", "kind": "creek", "name": "Islais Creek",
"lat": 37.7483, "lon": -122.376,
"same_as": {"wikidata": "Q6081595"},
"claims_columns": ["when", "property", "what", "outcome", "precision", "source", "link", "record_id", "from_tables"],
"claims": [["1857", "described in", "patent from the United States covered the bed of Islais Creek …", null, "area",
"Democracy's Library: Official opinions of Percy V. Long, City Attorney …",
"https://archive.org/details/C101744336/page/n163", "qipu:sf/place/creek/islais-creek", "x_dl_facts"], …],
"release": "v0.7.1", "licence": "CC0-1.0 (claims); sources keep their own rights"
}