Qipudata
San Francisco · v0.7.1
qipu data›San Francisco›Datasets›sources

Sources

sourcesrelease v0.7.1CC0 1.0

About

Every source cited. archive.org items carry their collections and Democracy's Library membership.

Where it comes from
The Internet Archive's own description of each item cited (title, collections, contributing library), and the address of the few web sources.
How it was read
Read from archive.org's metadata for each item.
How far to trust it
The release passed its automatic checks before it was published (what is checked). A hand check by the editor of qipu data is under way: 36 of 38 items checked so far were right, on votes, money, places and the map. A row says what the record states; the scanned page is the authority. Details
Licence
CC0 1.0: use it for anything, no permission needed. The scans and other sources keep their own rights.
Version
v0.7.1, published 21 October 2026. A published release is never changed; corrections come as a new release.
Technical details
Table
sources
Rows
6,377
Columns
20
Parquet
sources.parquet, 0.3 MB
Version
v0.7.1, 21 October 2026
Licence
CC0 1.0. Sources keep their own rights.

Rows

Rows by dl_rule

ValueRowsShare
(empty)3,07048.1%
no collection in DL scope (DL_INVENTORY.md section 1)2,60940.9%
collection in Democracy's Library scope: igscalocalgovdocs,instituteofgovernmentalstudieslibrary4316.8%
collection in Democracy's Library scope: democracys-library,igscalocalgovdocs,instituteofgovernmentalstudieslibrary2063.2%
collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_san-francisco-real-prop-report,instituteofgovernmentalstudieslibrary510.8%
collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_sfbcdc-minor-permits-granted,instituteofgovernmentalstudieslibrary40.1%
collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_annual-report-sf-public-utilities,instituteofgovernmentalstudieslibrary2< 0.1%
collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_sf-bos-annual-salary-ord,instituteofgovernmentalstudieslibrary2< 0.1%
collection in Democracy's Library scope: USGovernmentDocuments,us-supreme-court1< 0.1%
collection in Democracy's Library scope: nasa1< 0.1%

Rows by kind

ValueRowsShare
photo1,36421.4%
book1,06116.6%
web74311.7%
encyclopedia6309.9%
film6179.7%
audio5608.8%
board_minutes4777.5%
report2574.0%
plan1933.0%
board_journal1442.3%
council_minutes1352.1%
eir1171.8%

Rows by holder

ValueRowsShare
(empty)4,23366.4%
San Francisco Public Library90414.2%
Institute of Governmental Studies Library, University of California Berkeley69310.9%
SFGTV2183.4%
A collaborative of San Francisco neighborhood history groups1031.6%
San Francisco History Center, San Francisco Public Library791.2%
Internet Archive611.0%
CNNW320.5%
MSNBCW70.1%
KTVU70.1%
San Francisco Maritime National Historical Park3< 0.1%
Environmental Design Library, University of California Berkeley3< 0.1%

Method and accuracy

Read from archive.org's metadata for each item.

FieldMade byHow well it is known
sources.in_democracys_libraryrule: the item's collections vs DL scope (DL_INVENTORY.md)exact by definition, as of the metadata fetched 14/10/2026.

The release passed its automatic checks before it was published (what is checked). A hand check by the editor of qipu data is under way: 36 of 38 items checked so far were right, on votes, money, places and the map. A row says what the record states; the scanned page is the authority.

Methodology · Accuracy

Columns 20

ColumnTypeCan be emptyPrivacyPublic copyDescription
source_idVARCHARnononeas isThe source: an archive.org identifier, or web:<url> for other sources.
Can be empty no · Privacy none · Public copy as is
collectionVARCHARyesnoneas isFirst archive.org collection of the item.
Can be empty yes · Privacy none · Public copy as is
collectionsVARCHAR[]yesnoneas isAll archive.org collections of the item.
Can be empty yes · Privacy none · Public copy as is
in_democracys_libraryBOOLEANyesnoneas isTrue when one of the item's collections is in Democracy's Library scope: the DL pools (instituteofgovernmentalstudieslibrary, USGovernmentDocuments, fedlink, governmentpublications, us_census, nasa, ...) or one of the 830 child collections of democracys-library (DL_INVENTORY.md section 1). SF Public Library scans are not in DL.
Can be empty yes · Privacy none · Public copy as is
dl_ruleVARCHARyesnoneas isWhich collection made it DL, or why not.
Values collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_annual-report-sf-public-utilities,instituteofgovernmentalstudieslibrary, collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_san-francisco-real-prop-report,instituteofgovernmentalstudieslibrary, collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_sf-bos-annual-salary-ord,instituteofgovernmentalstudieslibrary, collection in Democracy's Library scope: USGovernmentDocuments,democracys-library,igscalocalgovdocs,igsl_sfbcdc-minor-permits-granted,instituteofgovernmentalstudieslibrary, collection in Democracy's Library scope: USGovernmentDocuments,us-supreme-court, collection in Democracy's Library scope: democracys-library,igscalocalgovdocs,instituteofgovernmentalstudieslibrary, collection in Democracy's Library scope: igscalocalgovdocs,instituteofgovernmentalstudieslibrary, collection in Democracy's Library scope: nasa, no collection in DL scope (DL_INVENTORY.md section 1)
Can be empty yes · Privacy none · Public copy as is
kindVARCHARnononeas isboard_journal, board_minutes, council_minutes, report, plan, eir, book, film, audio, photo, open_data, encyclopedia, map, web ...
Values appraisal, audio, board_journal, board_minutes, book, collection, council_minutes, directory, eir, election_results, encyclopedia, film ...
Can be empty no · Privacy none · Public copy as is
titleVARCHARyescatalogue textas isTitle from archive.org, else a citing claim's source label.
Can be empty yes · Privacy catalogue text · Public copy as is
creatorVARCHARyescatalogue textas isCreator as given by archive.org.
Can be empty yes · Privacy catalogue text · Public copy as is
publisherVARCHARyescatalogue textas isPublisher as given by archive.org.
Can be empty yes · Privacy catalogue text · Public copy as is
year_fromBIGINTyesnoneas isYear of the item (archive.org date), else the earliest year of the claims citing it.
Can be empty yes · Privacy none · Public copy as is
year_toBIGINTyesnoneas isLatest year of the claims citing it, else the item's year.
Can be empty yes · Privacy none · Public copy as is
pagesBIGINTyesnoneas isarchive.org imagecount.
Can be empty yes · Privacy none · Public copy as is
ocr_qualityVARCHARyesnoneas isThe OCR engine archive.org names (tesseract 5.3.0, ABBYY FineReader 9.0, ...); not a quality score.
Values ABBYY FineReader 11.0, ABBYY FineReader 11.0 (Extended OCR), ABBYY FineReader 8.0, ABBYY FineReader 9.0, tesseract 4.1.1, tesseract 5.0.0-1-g862e, tesseract 5.0.0-alpha-20201231-10-g1236, tesseract 5.0.0-beta-20210815, tesseract 5.0.0-rc1-12-g88b4, tesseract 5.0.0-rc2-1-gf788, tesseract 5.1.0-1-ge935, tesseract 5.2.0-1-gc42a ...
Can be empty yes · Privacy none · Public copy as is
licenceVARCHARyesnoneas isarchive.org licence URL or rights text (<= 300 characters); for web sources the citing claims' licence code.
Can be empty yes · Privacy none · Public copy as is
holderVARCHARyescatalogue textas isarchive.org contributor: the library that gave the copy (San Francisco Public Library, ...).
Can be empty yes · Privacy catalogue text · Public copy as is
digitizedBOOLEANyesnoneas isTrue for archive.org items; empty for web sources.
Can be empty yes · Privacy none · Public copy as is
on_archive_orgBOOLEANyesnoneas isTrue when the source is an archive.org item.
Can be empty yes · Privacy none · Public copy as is
urlVARCHARyesnoneas isLink to the source: the archive.org item page, or the web page.
Can be empty yes · Privacy none · Public copy as is
n_claimsDOUBLEyesnoneas isClaims citing it.
Can be empty yes · Privacy none · Public copy as is
notesVARCHARyesfree textprivate people's names withheld under the embargoEmpty in v0.4.
Can be empty yes · Privacy free text · Public copy private people's names withheld under the embargo

Privacy: what kind of text the column holds. Public copy: what the published file does with it; the embargo covers records after 1951.

Preview

source_idcollectioncollectionsin_democracys_librarydl_rulekindtitlecreator
potreroview10unse_3potreroview['potreroview', 'sanfrancisconewspapers', 'americana']falseno collection in DL scope (DL_INVENTORY.md section 1)bookThe Potrero View—
csf_000685sanfranciscopubliclibraryaudiovisual['sanfranciscopubliclibraryaudiovisual', 'californiarevealed', 'americana']falseno collection in DL scope (DL_INVENTORY.md section 1)bookHunter's Point Beacon 1945-07-20San Francisco Housing Authority
openpublica-san_francisco-california-bde5e4ad-fbe8-4f17-8a…open-publica['open-publica', 'community_media', 'newsandpublicaffairs']falseno collection in DL scope (DL_INVENTORY.md section 1)filmSan Francisco Police Commission Meeting – October 16, 2024OpenPublica
C124924210igscalocalgovdocs['igscalocalgovdocs', 'instituteofgovernmentalstudieslibrary', 'americana']truecollection in Democracy's Library scope: igscalocalgovdocs…bookSan Francisco 1970 census summary and analysis : Part II, …San Francisco (Calif.). Department of City Planning
C124877556igscalocalgovdocs['igscalocalgovdocs', 'instituteofgovernmentalstudieslibrary', 'americana']truecollection in Democracy's Library scope: igscalocalgovdocs…bookTranscript of hearing on the subject of central city probl…California. Legislature. Assembly. Interim Committee on Mu…
web:opensfhis…p33.04265.jpg——false—photoOpenSFHistory wnp33.04265—
web:commons.wikimedia.org/wiki/File:Historic_American_Buil…——false—photoA. J. Wittlock, Historic American Buildings Survey (Librar…—
web:opensfhis…np26.1362.jpg——false—photoOpenSFHistory wnp26.1362—

8 of 20 columns shown. Eight rows sampled at build time. Hover a shortened value for all of it.

Download

FormatFileSize
Parquetsources.parquet ↗0.3 MB
SQLite (all tables)qipu_sf.db ↗6.44 GB

SQL

SELECT * FROM sources LIMIT 100

Run in the SQL console (DuckDB in the browser, over the Parquet file).

Python

import duckdb duckdb.sql("SELECT * FROM 'https://tiles.qipu.org/data/releases/v0.7.1/parquet/sources.parquet' LIMIT 5").show()