Digital One Foundation The Text Layer
Paper 001 · Method

What the model reads.

A language model handed a PDF is not handed its text, and on Claude it is billed twice over for the same page: a page pays twice. These notes measure what conversion to Markdown removes — sixteen public documents, 861 pages, counted with the counter the claim is about — and set out the defect that made an earlier measurement wrong: the honest number moved downwards.

§1A page pays twice

A PDF is a set of drawing instructions. The text a reader sees is one layer of it — sometimes present, sometimes not, never the whole — which is why a scanned page and a born-digital page cost the same picture and different text, and why the corpus in §3 holds both. Before any of an estate reaches a model one decision comes first: hand the files over as they are, or convert them. The first is what the vendor's own upload exists for.

On Claude that means a document content block carrying the raw bytes. Two sentences of the vendor's description decide the bill: "The system converts each page of the document into an image", and "The text from each page is extracted and provided alongside each page's image".[1] The page arrives twice — as a picture of itself, and as the text lifted off it — and both are input tokens.

That is a feature before it is a cost — a model shown the picture can be asked about the chart on it — and a bill paid on every pass. The same documentation puts the text alone at 1,500–3,000 tokens a page, depending on content density, with the image cost added on top.[1]

The comparison every figure is stated against

Native document upload on Claude: a PDF sent as a document block, charged as page image plus extracted text. Every figure below is stated against that baseline, on the model it names. §7 names the vendor where the sign reverses.

§2Markdown is larger than a text dump

A paper that opened with the saving would be selling, so start with the unflattering number. Over the same sixteen documents the Markdown this engine produces costs 1,323 tokens a page against 1,239 for a plain text dump — 1.068×, rendered 1.07×. Per document it runs from 0.94× to 1.73×, and the 1.73× is a tax form, where the syntax that rebuilds a grid costs more than the words in it.

So against plain extracted text there is no saving to claim, and none is claimed. What the syntax buys is the grid, the heading tree, the numbering and the lists — a value stays in the row it was printed in. A number that has escaped its row is not a cheaper answer. It is a wrong one that costs less.

For much of an estate the comparison does not arise, because there is no native path to compare against. "Binary formats such as .xlsx or .docx are not supported in document blocks and must be converted to text or PDF first."[1] PPTX is absent from the types the Files API gives a document block.[2] There the question is not whether to convert but whether conversion keeps the structure.

§3The measurement, done properly

The corpus is not a sample of anyone's estate: sixteen public PDFs, 861 pages, weighted towards documents hard in different ways and dense-text-heavy, because dense text is where the Markdown is largest and the saving smallest. A corpus of slide decks and scanned forms would have flattered the engine. This one does not.

Each document is named in a manifest by publisher URL, sha256 and page count, and fetched from its publisher rather than redistributed.[3] Each was counted twice with messages.count_tokens — free, and called an estimate by the vendor itself: a count, not an invoice.[4] Once as a document block holding the raw PDF, once as a text block holding the Markdown crunch 0.1.0 made from it, scaffolding subtracted — each on Claude Opus 5, Claude Sonnet 5 and Claude Fable 5.1. The three returned identical counts. Not close: identical.

Totals are authoritative rather than rounded per-page figures: 1 − 1,139,102 ÷ 2,404,262 = 52.6%, rendered 53%. The mean is published rather than the median of 55.4%, because it is the lower of the two. Native costs 2,792 tokens a page and the Markdown 1,323, and the range runs from 11.1% to 73.1%: a mean over an arbitrary corpus is not a benchmark result, so the spread travels with it.

Document What it is Pages Native → crunched, tokens/page Fewer
nasa-tm-x-3181Scanned typewritten memo232,095 → 56473.1%
scotus-brownScanned court pages142,342 → 74568.2%
nist-csf2Security framework322,287 → 73367.9%
rfc9110Protocol specification1942,405 → 87663.6%
nist-sp800-63bTechnical guideline802,425 → 88863.4%
statute-79-pg437Scanned session law102,814 → 1,13859.6%
eurlex-gdpr-enEU regulation, English882,792 → 1,13659.3%
fed-h41Statistical release112,602 → 1,13556.4%
bis-quarterlyFinancial review, charts1022,532 → 1,15554.4%
cfr-40-1-1Regulation, one sparse page13,007 → 1,42752.5%
arxiv-gpt3Research paper, wide tables752,784 → 1,34551.7%
eurlex-gdpr-huThe same, in Hungarian883,605 → 1,94146.2%
arxiv-attentionMathematics-heavy paper152,557 → 1,41544.6%
arxiv-resnetTwo-column paper123,562 → 2,19738.3%
irs-i1040gi-2023Tax booklet, three columns1143,614 → 2,48331.3%
irs-f1040-2023Tax form on a grid23,320 → 2,95211.1%
CorpusTotals 2,404,262 → 1,139,1028612,792 → 1,32352.6%

§4A flat charge divided by a variable one

Of those 2,792 tokens about 1,568 are the page image, and that had to be established rather than looked up. The vendor publishes two tiers — standard, capped at 1,568 visual tokens an image, and high-resolution, for its 4.7-and-later models, capped at 4,784 — and says pages are rasterised server-side at dimensions you do not control.[5] The three models of §3 are 4.7-and-later models, so the table alone would place them on the higher cap, a factor near three; Claude Haiku 4.5, measured alongside them below, sits on the standard one.

The cheapest experiment closes it. One corpus document is a single page, and its whole native cost is 3,007 tokens, text and image together. A page cannot cost less than its own image, so 4,784 is impossible. Subtraction — native minus a plain pdfplumber text dump of the same PDF, no Markdown syntax, per page — puts the median at 1,569 tokens on Claude Opus 5 over all sixteen, range 1,309 to 1,618, and 1,568 on Claude Haiku 4.5 over the thirteen it accepted, range 1,289 to 1,616: one token apart across a supposed threefold boundary. The dump is a proxy for the vendor's own extractor, and where it over-counts, as on dense mathematics, the residual is understated — a bias running away from 4,784, not towards it. At 1,568 the extracted text implied by the corpus is 1,224 tokens a page; the dump came to 1,239, 1.2% apart.

Which is why the reduction column in §3 is mostly arithmetic: what conversion removes is the page image, and the page image does not care what is printed on it. Thirteen of the sixteen save between 1,365 and 1,676 tokens a page — inside 15% of the 1,568-token page image — while their reductions run from 38.3% to 73.1%. The three that save less save 368 to 1,141: a mathematics-heavy paper, a tax booklet, and the two-page tax form whose 11.1% is the floor. Across the corpus 1,469 tokens a page stop being billed. Estimate an estate in tokens, not in percent — what may not be said is that a particular estate will see 1,469.

MIXED PAGES EU regulation, English · 88 pages · 59.3% native · 2,792 tokens/page image 1,613 text 1,179 crunched · 1,136 tokens/page markdown saved 1,656 1,656 tokens/page stop being billed FIELD LABELS ON A GRID tax form · 2 pages · 11.1% native · 3,320 tokens/page image 1,618 text 1,702 crunched · 2,952 tokens/page markdown saved 368 368 tokens/page stop being billed the page image is the part that stops being billed, and it is the same size on both pages where the Markdown costs more than the text it replaces, most of that saving goes back
Figure 1 One page, two bills. Bars are tokens per page at one scale; the text figure is the measured plain-text dump, and the image figure is the remainder after it.

One pair makes the point without interpretation. The GDPR is published by the same office in English and in Hungarian — the same 88 pages, the same page image to within three tokens, 1,613 against 1,610. The Hungarian text costs 1.69× the English, the Markdown 1.71×; the reductions are 59.3% and 46.2%, and the tokens saved are nine apart. Thirteen points worse as a percentage, the same conversion in tokens: the difference is a property of the tokenizer, and it moves both sides together. The Hungarian conversion is not worse. The denominator is larger.

§5The defect: a claim about one vendor, counted with another's tokenizer

The method in §3 is the corrected one. The figure crunch.md published until 7 September 2026 was 64% — 64.4%, rendered 64%. It was obtained in the ordinary way: a corpus, a tokenizer, an arithmetic. Each part was defensible alone. Two were wrong together.

Defect · found by re-measuring with the vendor's own counter, 7 September 2026

A proxy tokenizer, and a corpus nobody outside could re-run

The conditions are ordinary rather than exotic. The counter was tiktoken o200k_base, another vendor's tokenizer, running locally. The corpus was sixteen private PDFs, 152 pages, on one laptop, and nobody outside could re-run it. The page image was read off a documentation table.

as published · 16 private PDFs · 152 pages · one laptop
counter   tiktoken.get_encoding("o200k_base")
native    566 text + 1,568 image = 2,134   image read off a cap
markdown  760
1 − 760 ÷ 2,134 = 64.4%   →   published as "64%"

re-counted · count_tokens · 16 public PDFs · 861 pages
1 − 1,139,102 ÷ 2,404,262 = 52.6%   →   published as "53%"

Two things changed at once. A proxy tokenizer does not fail loudly; it fails plausibly, returning numbers of the right order. And the private corpus was far sparser — part of why its extracted text came out at 566 tokens a page against the 1,224 the public corpus implies, and how much of the rest belongs to the change of counter cannot be recovered now. A private corpus also removed the mechanism that would have caught either: somebody outside re-running it.

The native side moved in the claim's favour; the engine's own output moved further against it. The expectation had been that 64% was conservative. The honest number goes down, not up. The vendor's own documentation says its 4.7-and-later models produce roughly 30% more tokens for the same text than its earlier ones[4] — the size of the error a stale count of its own makes; no published figure relates another vendor's tokenizer to this one, and it was never a stand-in.

A constant that already contained the words

The engine also reports a ratio per conversion, and its estimate of the native cost was the document's own extracted text plus a per-page constant of 1,750 — read as the mid-point of the 1,500–3,000 tokens a page the vendor quotes. The vendor labels that range text and adds the image on top, so the document's words were charged twice.

as shipped · the constant read as "text + page image"
native_estimate = extracted_text_tokens + 1,750 × pages
the document's words are counted once in the first term and again inside the second

corrected · the constant is the page image alone
native_estimate = extracted_text_tokens + 1,568 × pages
one 14-page document · reported ratio 0.688 → 0.657
a private file, counted with the engine's heuristic tokenizer — illustrative only

The documentation is what hid it: a reader checking the line sees a mid-point, a range and an attribution, and reads softness rather than error. A softer estimate would have been forgivable. Double-counting was arithmetic.

A score that fell because detection improved

The engine's other number is a structure score out of 100, and a repair to table detection made it fall.

one partially-ruled table, alternate rows shaded
before   6 one-row lattices; any under 2 rows dropped → 5 lost as prose
after    stitched back into one table
         40 markdown table lines → 89 · 0 words lost · 9 gained
         needs_help       23 → 23    integrity did not move
         structure_score  82 → 78

The conversion had plainly got better — nine words recovered, none lost, five discarded fragments stitched back into the one table they came from. The fall sat in the tables component: a metric is defined only over what the pipeline detected, so what it missed is not scored badly — it is not scored at all. Twice in one session, the same night 64% became 53%, a number got worse because the thing measuring it had got honest.

§6Where it runs, and what the result records

The base conversion path is deterministic by design and by documentation — the same bytes in, the same Markdown out — and it calls no model. An enhancement tier that would call one is planned and does not exist in the code these notes draw on; the property tests that would hold determinism there are a plan rather than a suite, and there are no golden files to diff against.

Across that path one module opens a socket: the fetcher that exists because a caller asked for a URL. One more can — an optional token counter, off by default, needing both a credential and a flag. Naming both is the point. OCR is where a page most obviously could leave, so it ships as a provider protocol with none enabled by default: an installed provider is listed rather than activated, and with nothing named a scan is detected, its pages are named, and the conversion refuses rather than inventing text. Egress is a declared field — local, private_network or third_party — checked before the first page is rendered and recorded on the result, so a cloud provider is turned away before it authenticates.

The shipped provider is local Tesseract with seven language packs (hu, de, en, pl, ro, cs, sk), handed rasters rather than the document; a converter for legacy Office formats declares a risk rung the same way but is handed the file itself, and nothing in its sandbox stops that program opening a socket. The protocol cannot prove a provider opens no socket and does not pretend to: it makes the claim explicit, checkable in advance and recorded.

What conversion buys at the far end is one context, and therefore one answer. RFC 9110 is 194 pages;[6] converted and counted on Claude Haiku 4.5, 126,739 tokens — inside that model's 200K window, the smallest in the range.[7] Three corpus documents were refused outright as native uploads there: a maximum of 100 PDF pages may be provided. All three counted as Markdown, and two fit the window they had been refused from. The third does not. The window decides what fits, not a page count, and the engine caps intake at 2,000 pages — "the longest thing in the estate goes in whole" is not a sentence this paper can write.

Throughput is CPU-bound and per core — about 6 ms a page for a report with a clean text layer, about 513 ms for a dense design-heavy one, median of three runs on one production machine. Neither document is in the public corpus, so these two figures cannot be re-run from this paper. The 86× is the document, not the machine, and there is no GPU in the path: a machine full of accelerators converts an estate with them idle.

§7What this cannot do, stated before anyone else says it

A paper that lists only what a conversion keeps is marketing with a token count attached. These losses are structural rather than temporary:

  1. A merged cell is flattened. The GitHub Flavored Markdown table extension defines a header row, a delimiter row and data rows; colspan, rowspan and merged cell do not occur in the specification.[8] In a PDF a spanning cell keeps its text in the first column it covered and the rest come through blank; in a presentation the covered cells are gone rather than blank. The table is named in a warning, the structure score subtracts, and no completeness claim about tables survives it.
  2. A picture is not read. A figure becomes a placeholder holding its position, carrying a caption only where one was found on the page — never one generated on the page's behalf. The dimensions citable out of a drawing are therefore the ones that were in a table, and because a merged cell there is flattened, not even all of those.
  3. Some structure is not extracted at all. DOCX headers, footers, footnote and endnote text are not, and the emitter has no footnote syntax on any input path — so nothing here claims footnotes survive as footnotes. Tables are skipped on rotated pages; formula cells with no cached result come through empty. Most of those announce themselves in a warning. The missing footnote syntax does not.
  4. A line read off pixels is capped below certain — never certain, and flagged for help at or below guessed — and handwriting is declined rather than guessed: a signature or a margin note comes back as a page named for a person to check rather than as text.
  5. On Gemini the sign reverses. Google's documentation states that each document page is equivalent to 258 tokens, and that you are not charged for tokens originating from the extracted native text in PDFs.[9] Markdown at 1,323 tokens a page therefore costs about 5.1× the native file there. On Gemini, send the PDF.

None of these is a reason not to convert. They are the reasons to describe the conversion accurately.

§8What is open past the text layer

Everything above describes a floor rather than a finished thing. Extraction from a text layer is deterministic and checkable — a value can be traced to the row it was printed in — while anything said about a picture is generated, and is a claim. Three questions sit past that line and none is answered here: how a description of a figure might reach a model honestly, labelled as derived, kept below certain, carrying a way back to the pixels; where converted text should live once there is a corpus of it, given that the engine returns Markdown and a structure map and keeps neither, by decision rather than omission; and what an estate's own repetition makes compressible. They are the subject of Paper 002, marked there as open research rather than as results. None of them exists in the code these notes draw on.

On the provenance of this material

The method and measurements here were developed inside crunch.md, a Digital One product, and the defect in §5 was found by re-measuring that product's own published claim. The engine, crunch-core, and its MCP server are Apache-2.0 licensed and run offline; the hosted service is proprietary. The repository is not public yet, so that licence is a statement of intent rather than something a reader can check today. The corpus is public, and the native column can be re-counted by anyone holding the manifest and a key. Deliberately absent: prices, plans and tiers, and deployment internals.

References

  1. Anthropic, PDF support, "How PDF support works" and "Estimate your costs"; the binary-formats sentence quoted in §2 is on this page, not the Files API page. platform.claude.com/docs/en/build-with-claude/pdf-support, read 11 September 2026.
  2. Anthropic, Files API, "File type support"; PPTX is absent from the supported-types table. platform.claude.com/docs/en/build-with-claude/files, read 11 September 2026.
  3. The corpus. It is named in a manifest — packages/crunch-core/tests/corpus/manifest.toml, sha256 f4abe7ec1bc2ffa8b499508810bec3e4c5009ddbcd83c3655e3d36c7242cbaf0 — which sits in a repository that is not public yet, so its rows are printed here instead: id, publisher URL, and the sha256 of the file as published on 7 September 2026. A publisher who re-issues a document changes the hash, and a re-run should say so rather than measure a different document under the same name. The two EUR-Lex documents answer an automated fetch with a bot challenge and must be fetched by hand.
    arxiv-attention   https://arxiv.org/pdf/1706.03762v7
                      bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697
    arxiv-resnet      https://arxiv.org/pdf/1512.03385v1
                      1e0651b6810ecba34a3dbc5b5b0209226f889004607c1f203540a48d64e5a93a
    arxiv-gpt3        https://arxiv.org/pdf/2005.14165v4
                      97fd272f1fdfc18677462d0292f5fbf26ca86b4d1b485c2dba03269b643a0e83
    nist-csf2         https://nvlpubs.nist.gov/nistpubs/CSWP/NIST.CSWP.29.pdf
                      3c31f46fee98cac0c4323453e5109291a213b4de7fef8c058af9bf67f717433c
    nist-sp800-63b    https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-63b.pdf
                      ccfce7510a1267933f912d023306c6e7d485f21be63d271108b61aad6139127e
    rfc9110           https://www.rfc-editor.org/rfc/rfc9110.pdf
                      60b30efa1048900833d1758440247fe8ac85a3134f2327388dcb24e07d814c89
    eurlex-gdpr-en    https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
                      bd84e63f5b622b739a83389afc3b30d240f792bb88d8eb03a816c9a82b0c2499
    eurlex-gdpr-hu    https://eur-lex.europa.eu/legal-content/HU/TXT/PDF/?uri=CELEX:32016R0679
                      b4671fe07338c3b4b346128cf3a7652c8d57f341deb512ec9bc7c74f30074fa8
    cfr-40-1-1        https://www.govinfo.gov/content/pkg/CFR-2023-title40-vol1/pdf/CFR-2023-title40-vol1-sec1-1.pdf
                      67fefda601faab628481ada17684bb65e3392e2e9db879da5276291ba3f2c7e1
    irs-f1040-2023    https://www.irs.gov/pub/irs-prior/f1040--2023.pdf
                      a410ce39e1f9b75ab3ddd8c314b9e7dce022ed6e3aebde5418a58b88f1e0213c
    irs-i1040gi-2023  https://www.irs.gov/pub/irs-prior/i1040gi--2023.pdf
                      90c8759c7a829d8c74296187cdbdb47bc9025480fffc41ff5e7225889c8fea7e
    fed-h41           https://www.federalreserve.gov/releases/h41/20240104/h41.pdf
                      614f7600fcde1ba0ec1db2047580a518fe2888ecc7808e76aea7e27e5a7fa6ba
    bis-quarterly     https://www.bis.org/publ/qtrpdf/r_qt2403.pdf
                      ba5a32c4649e9202733d0ea83f506b43ad689b81fd23bf802a1d805681cb70cb
    scotus-brown      https://tile.loc.gov/storage-services/service/ll/usrep/usrep347/usrep347483/usrep347483.pdf
                      7021074961cffdafdddac07c25c6b21c6a689d1ff118966f5b53bb3a48163ec1
    statute-79-pg437  https://www.govinfo.gov/content/pkg/STATUTE-79/pdf/STATUTE-79-Pg437.pdf
                      67439f5db51cb49a6cd2423c28c257a7fdc1112ccd6f605ab4595eebbbcc53c5
    nasa-tm-x-3181    https://ntrs.nasa.gov/api/citations/19750007805/downloads/19750007805.pdf
                      0135848ec946da34297f79fae489b54b9d9b1ace545e187d333b0dfb47a6b324
  4. Anthropic, Token counting, "Pricing and rate limits" and the tokenizer note for Claude 4.7 and later models. platform.claude.com/docs/en/build-with-claude/token-counting, read 11 September 2026. The identical counts across three models are this paper's measurement, not a vendor statement.
  5. Anthropic, Vision — Resolution and token cost, platform.claude.com/docs/en/build-with-claude/vision, and Coordinates and bounding boxes, platform.claude.com/docs/en/build-with-claude/vision-coordinates, both read 11 September 2026. Neither states what size a PDF page is rendered at, which is why §4 measures it.
  6. R. Fielding, M. Nottingham, J. Reschke (eds), HTTP Semantics, RFC 9110 (STD 97), June 2022, 194 pp. The artefact measured is the RFC Editor's PDF rendering, sha256 60b30efa1048900833d1758440247fe8ac85a3134f2327388dcb24e07d814c89. rfc-editor.org/rfc/rfc9110.pdf, read 11 September 2026.
  7. Anthropic, Models overview, "Compare models" — context windows of 1M tokens for Claude Fable 5.1, Claude Opus 5 and Claude Sonnet 5, and 200K for Claude Haiku 4.5. platform.claude.com/docs/en/about-claude/models/overview, read 11 September 2026.
  8. GitHub, GitHub Flavored Markdown Spec, version 0.29-gfm (2019-04-06), §4.10 "Tables (extension)"; colspan, rowspan and merged cell do not occur in it. github.github.com/gfm/, read 11 September 2026.
  9. Google, Document understanding — Technical details, Gemini API documentation. ai.google.dev/gemini-api/docs/document-processing, read 11 September 2026.