Digital One Foundation The Text Layer
Paper 001 · Method

What the model reads.

A language model handed a PDF is not handed its text, and on Claude it is billed twice over for the same page: a page pays twice. These notes measure what conversion to Markdown removes — sixteen public documents, 861 pages, counted with messages.count_tokens — and what it cannot keep.

§1A page pays twice

A PDF is a set of drawing instructions. The text a reader sees is one layer of it — sometimes present, sometimes not, never the whole — which is why a scanned page and a born-digital page cost the same picture and different text, and why the corpus in §3 holds both. Before any of an estate reaches a model one decision comes first: hand the files over as they are, or convert them. The first is what the vendor's own upload exists for.

On Claude that means a document content block carrying the raw bytes. Two sentences of the vendor's description decide the bill: "The system converts each page of the document into an image", and "The text from each page is extracted and provided alongside each page's image".[1] The page arrives twice — as a picture of itself, and as the text lifted off it — and both are input tokens.

That is a feature before it is a cost — a model shown the picture can be asked about the chart on it — and a bill paid on every pass. The same documentation puts the text alone at 1,500–3,000 tokens a page, depending on content density, with the image cost added on top.[1]

The comparison every figure is stated against

Native document upload on Claude: a PDF sent as a document block, charged as page image plus extracted text. Every figure below is stated against that baseline, on the model it names. §4 names the vendor where the sign reverses.

§2Markdown is larger than a text dump

A paper that opened with the saving would be selling, so start with the unflattering number. Over the same sixteen documents the Markdown this engine produces costs 1,323 tokens a page against 1,239 for a plain text dump — 1.068×, rendered 1.07×. Per document it runs from 0.94× to 1.73×, and the 1.73× is a tax form, where the syntax that rebuilds a grid costs more than the words in it.

So against plain extracted text there is no saving to claim, and none is claimed. What the syntax buys is the grid, the heading tree, the numbering and the lists — a value stays in the row it was printed in. A number that has escaped its row is not a cheaper answer. It is a wrong one that costs less.

For much of an estate the comparison does not arise, because there is no native path to compare against. "Binary formats such as .xlsx or .docx are not supported in document blocks and must be converted to text or PDF first."[1] PPTX is absent from the types the Files API gives a document block.[2] There the question is not whether to convert but whether conversion keeps the structure.

§3The measurement, done properly

The corpus is not a sample of anyone's estate: sixteen public PDFs, 861 pages, weighted towards documents hard in different ways and dense-text-heavy, because dense text is where the Markdown is largest and the saving smallest. A corpus of slide decks and scanned forms would have flattered the engine. This one does not.

Each document is named in a manifest by publisher URL, sha256 and page count, and fetched from its publisher rather than redistributed.[3] Each was counted twice with messages.count_tokens — free, and called an estimate by the vendor itself: a count, not an invoice.[4] Once as a document block holding the raw PDF, once as a text block holding the Markdown crunch 0.1.0 made from it, scaffolding subtracted — each on Claude Opus 5, Claude Sonnet 5 and Claude Fable 5.1. The three returned identical counts. Not close: identical.

Totals are authoritative rather than rounded per-page figures: 1 − 1,139,102 ÷ 2,404,262 = 52.6%, rendered 53%. The mean is published rather than the median of 55.4%, because it is the lower of the two. Native costs 2,792 tokens a page and the Markdown 1,323, and the range runs from 11.1% to 73.1%: a mean over an arbitrary corpus is not a benchmark result, so the spread travels with it.

Document What it is Pages Native → crunched, tokens/page Fewer
nasa-tm-x-3181Scanned typewritten memo232,095 → 56473.1%
scotus-brownScanned court pages142,342 → 74568.2%
nist-csf2Security framework322,287 → 73367.9%
rfc9110Protocol specification1942,405 → 87663.6%
nist-sp800-63bTechnical guideline802,425 → 88863.4%
statute-79-pg437Scanned session law102,814 → 1,13859.6%
eurlex-gdpr-enEU regulation, English882,792 → 1,13659.3%
fed-h41Statistical release112,602 → 1,13556.4%
bis-quarterlyFinancial review, charts1022,532 → 1,15554.4%
cfr-40-1-1Regulation, one sparse page13,007 → 1,42752.5%
arxiv-gpt3Research paper, wide tables752,784 → 1,34551.7%
eurlex-gdpr-huThe same, in Hungarian883,605 → 1,94146.2%
arxiv-attentionMathematics-heavy paper152,557 → 1,41544.6%
arxiv-resnetTwo-column paper123,562 → 2,19738.3%
irs-i1040gi-2023Tax booklet, three columns1143,614 → 2,48331.3%
irs-f1040-2023Tax form on a grid23,320 → 2,95211.1%
CorpusTotals 2,404,262 → 1,139,1028612,792 → 1,32352.6%

§4What this cannot do, stated before anyone else says it

A paper that lists only what a conversion keeps is marketing with a token count attached. These losses are structural rather than temporary:

  1. A merged cell is flattened. The GitHub Flavored Markdown table extension defines a header row, a delimiter row and data rows; colspan, rowspan and merged cell do not occur in the specification.[5] In a PDF a spanning cell keeps its text in the first column it covered and the rest come through blank; in a presentation the covered cells are gone rather than blank. The table is named in a warning, the structure score subtracts, and no completeness claim about tables survives it.
  2. A picture is not read. A figure becomes a placeholder holding its position, carrying a caption only where one was found on the page — never one generated on the page's behalf. The dimensions citable out of a drawing are therefore the ones that were in a table, and because a merged cell there is flattened, not even all of those.
  3. Some structure is not extracted at all. DOCX headers, footers, footnote and endnote text are not, and the emitter has no footnote syntax on any input path — so nothing here claims footnotes survive as footnotes. Tables are skipped on rotated pages; formula cells with no cached result come through empty. Most of those announce themselves in a warning. The missing footnote syntax does not.
  4. A line read off pixels is capped below certain — never certain, and flagged for help at or below guessed — and handwriting is declined rather than guessed: a signature or a margin note comes back as a page named for a person to check rather than as text.
  5. On Gemini the sign reverses. Google's documentation states that each document page is equivalent to 258 tokens, and that you are not charged for tokens originating from the extracted native text in PDFs.[6] Markdown at 1,323 tokens a page therefore costs about 5.1× the native file there. On Gemini, send the PDF.

None of these is a reason not to convert. They are the reasons to describe the conversion accurately.

On the provenance of this material

The method and measurements here were developed inside crunch.md, a Digital One product; the defect found by re-measuring that product's own published claim is set out in Paper 002, §2. The engine, crunch-core, and its MCP server are Apache-2.0 licensed and run offline; the hosted service is proprietary. The repository is not public yet, so that licence is a statement of intent rather than something a reader can check today. The corpus is public, and the native column can be re-counted by anyone holding the manifest and a key. Deliberately absent: prices, plans and tiers, and deployment internals.

References

  1. Anthropic, PDF support, "How PDF support works" and "Estimate your costs"; the binary-formats sentence quoted in §2 is on this page, not the Files API page. platform.claude.com/docs/en/build-with-claude/pdf-support, read 11 September 2026.
  2. Anthropic, Files API, "File type support"; PPTX is absent from the supported-types table. platform.claude.com/docs/en/build-with-claude/files, read 11 September 2026.
  3. The corpus. It is named in a manifest — packages/crunch-core/tests/corpus/manifest.toml, sha256 f4abe7ec1bc2ffa8b499508810bec3e4c5009ddbcd83c3655e3d36c7242cbaf0 — which sits in a repository that is not public yet, so its rows are printed here instead: id, publisher URL, and the sha256 of the file as published on 7 September 2026. A publisher who re-issues a document changes the hash, and a re-run should say so rather than measure a different document under the same name. The two EUR-Lex documents answer an automated fetch with a bot challenge and must be fetched by hand.
    arxiv-attention   https://arxiv.org/pdf/1706.03762v7
                      bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697
    arxiv-resnet      https://arxiv.org/pdf/1512.03385v1
                      1e0651b6810ecba34a3dbc5b5b0209226f889004607c1f203540a48d64e5a93a
    arxiv-gpt3        https://arxiv.org/pdf/2005.14165v4
                      97fd272f1fdfc18677462d0292f5fbf26ca86b4d1b485c2dba03269b643a0e83
    nist-csf2         https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.29.pdf
                      3c31f46fee98cac0c4323453e5109291a213b4de7fef8c058af9bf67f717433c
    nist-sp800-63b    https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-63b.pdf
                      ccfce7510a1267933f912d023306c6e7d485f21be63d271108b61aad6139127e
    rfc9110           https://www.rfc-editor.org/rfc/rfc9110.pdf
                      60b30efa1048900833d1758440247fe8ac85a3134f2327388dcb24e07d814c89
    eurlex-gdpr-en    https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
                      bd84e63f5b622b739a83389afc3b30d240f792bb88d8eb03a816c9a82b0c2499
    eurlex-gdpr-hu    https://eur-lex.europa.eu/legal-content/HU/TXT/PDF/?uri=CELEX:32016R0679
                      b4671fe07338c3b4b346128cf3a7652c8d57f341deb512ec9bc7c74f30074fa8
    cfr-40-1-1        https://www.govinfo.gov/content/pkg/CFR-2023-title40-vol1/pdf/CFR-2023-title40-vol1-sec1-1.pdf
                      67fefda601faab628481ada17684bb65e3392e2e9db879da5276291ba3f2c7e1
    irs-f1040-2023    https://www.irs.gov/pub/irs-prior/f1040--2023.pdf
                      a410ce39e1f9b75ab3ddd8c314b9e7dce022ed6e3aebde5418a58b88f1e0213c
    irs-i1040gi-2023  https://www.irs.gov/pub/irs-prior/i1040gi--2023.pdf
                      90c8759c7a829d8c74296187cdbdb47bc9025480fffc41ff5e7225889c8fea7e
    fed-h41           https://www.federalreserve.gov/releases/h41/20240104/h41.pdf
                      614f7600fcde1ba0ec1db2047580a518fe2888ecc7808e76aea7e27e5a7fa6ba
    bis-quarterly     https://www.bis.org/publ/qtrpdf/r_qt2403.pdf
                      ba5a32c4649e9202733d0ea83f506b43ad689b81fd23bf802a1d805681cb70cb
    scotus-brown      https://tile.loc.gov/storage-services/service/ll/usrep/usrep347/usrep347483/usrep347483.pdf
                      7021074961cffdafdddac07c25c6b21c6a689d1ff118966f5b53bb3a48163ec1
    statute-79-pg437  https://www.govinfo.gov/content/pkg/STATUTE-79/pdf/STATUTE-79-Pg437.pdf
                      67439f5db51cb49a6cd2423c28c257a7fdc1112ccd6f605ab4595eebbbcc53c5
    nasa-tm-x-3181    https://ntrs.nasa.gov/api/citations/19750007805/downloads/19750007805.pdf
                      0135848ec946da34297f79fae489b54b9d9b1ace545e187d333b0dfb47a6b324
  4. Anthropic, Token counting, "Pricing and rate limits". platform.claude.com/docs/en/build-with-claude/token-counting, read 11 September 2026. The identical counts across three models are this paper's measurement, not a vendor statement.
  5. GitHub, GitHub Flavored Markdown Spec, version 0.29-gfm (2019-04-06), §4.10 "Tables (extension)"; colspan, rowspan and merged cell do not occur in it. github.github.com/gfm/, read 11 September 2026.
  6. Google, Document understanding — Technical details, Gemini API documentation. ai.google.dev/gemini-api/docs/document-processing, read 11 September 2026.