Pages
pagesrelease v0.7.1CC0 1.0
About
Every archive.org page cited. We point at the scans; we never copy them.
- Where it comes from
- The scans on the Internet Archive. A row is a page the data cites; the scans themselves are not copied.
- How it was read
- Each page of the scanned text is matched to its page in the archive.org viewer with the volume's page map; page_exact is false where the link may be a page or two off.
- How far to trust it
- The release passed its automatic checks before it was published (what is checked). A hand check by the editor of qipu data is under way: 36 of 38 items checked so far were right, on votes, money, places and the map. A row says what the record states; the scanned page is the authority. Details
- Licence
- CC0 1.0: use it for anything, no permission needed. The scans and other sources keep their own rights.
- Version
- v0.7.1, published 21 October 2026. A published release is never changed; corrections come as a new release.
Technical details
- Table
- pages
- Rows
- 183,888
- Columns
- 14
- Parquet
- pages.parquet, 6.8 MB
- Version
- v0.7.1, 21 October 2026
- Licence
- CC0 1.0. Sources keep their own rights.
Rows
Rows by ocr_engine
| Value | Rows | Share | |
|---|---|---|---|
| (empty) | 134,074 | 72.9% | |
| tesseract 5.3.0-6-g76ae | 38,041 | 20.7% | |
| ABBYY FineReader 9.0 | 7,326 | 4.0% | |
| ABBYY FineReader 8.0 | 4,447 | 2.4% |
Method and accuracy
Each page of the scanned text is matched to its page in the archive.org viewer with the volume's page map; page_exact is false where the link may be a page or two off.
| Field | Made by | How well it is known |
|---|---|---|
| pages.display_n / page links | per-volume leaf map: the audited Journal map, IA scandata for the other items (qipu_data.leafmap_ia) | 179,043 of 180,775 pages are mapped OCR page -> viewer page (page_exact); the rest keep n = OCR page and may be a page or two off. iiif_url is $<display_n + 1> (that IIIF service counts the shown pages from 1). Gate L1 checks the numbering; L2 checks that a fixed sample of quotes is on the cited page. |
The release passed its automatic checks before it was published (what is checked). A hand check by the editor of qipu data is under way: 36 of 38 items checked so far were right, on votes, money, places and the map. A row says what the record states; the scanned page is the authority.
Columns 14
| Column | Type | Can be empty | Privacy | Public copy | Description |
|---|---|---|---|---|---|
| page_id | VARCHAR | no | none | as is | The archive.org page: <identifier>:<leafNum> (IA's leaf number, the jp2 / IIIF number), or <identifier>:d<n> where only the BookReader index is known, or <url>:<page> for a web document. nopage:<seq> where the source gives no page number (sfbos rows without one): the row is kept, page_url is NULL.Values e.g. journalofproceed30sanfrich:1 |
| page_url | VARCHAR | yes | none | as is | Opens the page: archive.org BookReader link built from the leaf map (audited against IA's scandata), or <pdf url>#page=<n> for a PDF. NULL only where page_id is nopage:<seq>.Values e.g. https://archive.org/details/journalofproceed30sanfrich/page/n1 |
| source_id | VARCHAR | no | none | as is | archive.org identifier of the volume. |
| printed_page | VARCHAR | yes | none | as is | The page number printed on the page, from IA's page_numbers file (empty where IA has none, or where its page list does not match the OCR). |
| meeting_date | DATE | yes | none | as is | Earliest dated claim on the page. |
| text_sha1 | VARCHAR | yes | none | as is | Empty in v1. |
| ocr_engine | VARCHAR | yes | none | as is | Empty in v1. Values ABBYY FineReader 8.0, ABBYY FineReader 9.0, tesseract 5.3.0-6-g76ae |
| scan_url | VARCHAR | yes | none | as is | Opens the page on archive.org. |
| iiif_url | VARCHAR | yes | none | as is | The page image through archive.org IIIF (v3). Known fault: on about a third of pages it points one page early (an old IIIF service counted from 1); use page_url (the audited viewer page) until v0.6 fixes it. |
| leaf_num | INTEGER | yes | none | as is | The page's OCR leaf (the jp2/IIIF file number), from the volume's leaf map; empty in volumes without one. |
| display_n | INTEGER | yes | none | as is | archive.org BookReader page index (the n in /page/n<k>). |
| page_exact | BOOLEAN | yes | none | as is | True when the archive.org page number was converted from the OCR leaf with the per-volume leaf map; false = may be a page or two off. |
| n_claims | BIGINT | yes | none | as is | Claims citing the page. |
| ocr_file_sha1 | VARCHAR | yes | none | as is | SHA-1 of the volume's _djvu.xml OCR file (see ocr_files); the same for every page of a volume. |
Privacy: what kind of text the column holds. Public copy: what the published file does with it; the embargo covers records after 1951.
Preview
8 of 14 columns shown. Eight rows sampled at build time. Hover a shortened value for all of it.
Download
| Format | File | Size |
|---|---|---|
| Parquet | pages.parquet ↗ | 6.8 MB |
| SQLite (all tables) | qipu_sf.db ↗ | 6.44 GB |
SQL
SELECT *
FROM pages
LIMIT 100
Run in the SQL console (DuckDB in the browser, over the Parquet file).
Python
import duckdb
duckdb.sql("SELECT * FROM 'https://tiles.qipu.org/data/releases/v0.7.1/parquet/pages.parquet' LIMIT 5").show()