Research · Dataset v1 · Timing addendum 1

OCR and field extraction on 24 synthetic Turkish documents

We ran two Turkish OCR models on the same 24 synthetic documents using identical field extraction rules. The larger model was not more accurate in this limited test—an example of why you should test your own data before judging by a model's name.

Results and methodology

What does this test compare?

OCR + fixed field extraction pipelines are compared; this is not a ranking of generative language models or complete document processing systems. OCR converts images to text; the same regular expression rules locate the document number, date, seller, currency, subtotal, VAT, and grand total. No human corrections or date punctuation repairs were applied.

In Tesseract's official model description fast and best target different speed/accuracy trade-offs. This test used the Turkish LSTM files from tag 4.1.0 of both repositories. Model size does not guarantee better results on every document set.

Initial study: field accuracy and processing time

Initial study v1: 24 documents × 7 fields = 168 fields per model. One timing measurement per document/model; milliseconds.
ModelCorrect fieldsMissingIncorrectFully correct documentsMedian msp95 ms
fast166/168 (%98,8)2022/24156,9189,4
best159/168 (%94,6)4518/24296,0345,9

The fast model produced 166 correct fields in this set; the best model produced 159. This does not mean the same ranking will hold for more complex real documents. An unreadable label and an incorrectly read numerical value are both errors for the end user and both are included in measurement.

6 documents × 7 fields per condition. Each cell shows correct/42.
Image conditionFastBest
Clean42/4242/42
Small text42/4242/42
Rotated by 3°40/4241/42
Low resolution42/4234/42

What types of errors occurred?

The fast model failed to find the date field in two rotated documents. The best model failed to find four VAT fields at low resolution; periods were read as commas in four dates, and one order number gained an extra letter. For example, 08.10.2026 being read as 08.10,2026 was counted as incorrect because our date specification required an exact match.

This distinction affects production design: do not rely on the same check for an empty field, a malformed value, and a plausible but incorrect value. Total reconciliation, required-field validation, and human approval where needed are designed separately before turning OCR output into a business record.

From the initial study's raw records; expected values were fixed before testing.
Document / modelExpected / OCR outputEvaluation
FAT-rotated-1 / fastTarih: 06.10.2026 → Tarih; 06.10.2026The date digits are correct; because the colon became a semicolon, the fixed label rule cannot find the field: missing.
FAT-low_resolution-1 / bestKDV: 302,15 → KOV: 302,15The value is correct but the label is wrong: the extracted VAT field is missing. The image reads KDV.
FAT-low_resolution-1 / best08.10.2026 → 08.10,2026The field was found; the changed date punctuation makes it incorrect.
SIP-rotated-1 / bestSIP-2026-0013 → SsIP-2026-0013Extra letter in the document number: incorrect. The amount 1734 ,25 in the same document was correctly normalized to 1734.25 using the predefined whitespace/decimal normalization.

The 168 expected fields across 24 documents were independently rechecked against the generation formulas and amount arithmetic; the 336 field evaluations in the initial study were recalculated. Error-containing images were compared with raw text. We measure the end-to-end extraction pipeline, including label detection, rather than OCR character accuracy alone. We did not change label tolerance afterwards to improve the old score; a different parser requires a separate experiment.

Test set and predefined ground truth

The documents were generated synthetically: three types headed invoice, order, and quotation; clean, small-text, 3° rotation, and low-resolution conditions; two examples per cell. These are variations of the same single-column label template. They do not represent 24 independent real document layouts. The seven ground-truth fields were written before execution to gold.json . The protocol and image hashes were fixed; the runner verifies them.

Synthetic invoice generated under low-resolution conditions: document number, date, seller, and total fields.
Example test document; not a real invoice, company, or customer record.

Conditions for reproducing the test

  • Environment: Apple M4, 10 logical cores, 16 GB memory, macOS/arm64, Node v24.17.0.
  • Engine: Tesseract.js 6.0.1; LSTM/OEM 1, page segmentation PSM 6, preservation of interword spaces enabled.
  • Each model was preloaded. One unmeasured warm-up was performed with the first document; all 24 documents were then measured once. Model order alternated for each document. There were no retries in the 48 measured operations.
  • Timing: OCR + fixed field extraction. Excludes disk reads, model downloading/loading, and warm-up. Loading and warm-up times are recorded separately in the raw log.
  • Text: Unicode NFC and excess-whitespace normalization. For monetary values, the decimal comma was converted to a period and values were compared to two decimal places. Other characters required an exact match. p95 is the 23rd of 24 sorted measurements.

During initial setup, bundled version 7.0.0 raised a missing-engine-symbol error before document measurements began. The measured run used 6.0.1. This setup error and all failed fields were retained in the record; results were not cherry-picked.

Timing addendum 1: six repetitions on the same documents

On 15 October 2026, 10:05–10:06 UTC, six rounds were run using the same device, engine, model files, 24 images, and extraction rules: 144 operations per model, 288 in total. These are not 144 different documents. The initial v1 results remain unchanged above; the timings below come from a separate experiment.

Post-warm-up OCR + field extraction; ms. n=144 for each model.
ModelMedianp95Median when firstMedian when secondRange of round medians
fast141,1153,3140,5141,2140,0–142,5
best272,5285,4272,8272,3271,9–273,8

Each document/model ran first three times and second three times; document order also alternated forward/backward. First–second order medians differed by approximately 0.5% for fast and 0.2% for best: the ranking did not reverse under these conditions. Field outputs matched v1 in all six rounds; these repetitions provide no new, independent document evidence for accuracy.

The models were loaded and warmed up once each; the same workers were retained for all six rounds. Image reading, hash checking, loading, warm-up, scoring, and disk writes were excluded from timing; OCR and unchanged field extraction were included. Using the nearest-rank method, p95 is the 137th of 144 measurements. The download includes round, order, raw text, and load records.

No other build or experiment was started during measurement; existing applications remained open. Activity across all cores, including OCR, was 16.1–18.9% by round; the one-minute system load average fell from 12.9 to 7.1. These metrics are different and do not separate background load from OCR. Round medians stayed within a narrow range; nevertheless, the new timings were lower than the initial overnight measurements. This experiment alone cannot determine the contribution of device load, caching, or session conditions. It does not establish guarantees for cold starts, heavy concurrency, or production latency.

What was the cost per document?

External API fee: 0 USD per document for both models (15 October 2026). Processing took place on a local computer, with no paid API calls. This is not total cost of ownership: hardware, electricity, development, maintenance, and human correction effort were not priced. No hypothetical savings amount was derived from CPU time.

What decision can these results support?

If your document flow is stable and labeled, local OCR with rule-based extraction may be a starting point for a pilot. This test is insufficient for decisions involving complex tables, handwriting, variable layouts, critical amounts, or interpretation across documents. Define your own representative documents, permissions, and acceptance criteria, then run a new comparison.

A small synthetic sample; the same template; one device; one measurement round in v1 and six in the timing addendum; an environment where background tasks were not fully stopped: these limit generalization of accuracy and timing. This is not a price quote, SLA, security assurance, or result representative of Türkiye as a whole.

Data, code, and responsibility

Download the 24 documents, expected ground truth, 48 outputs, and execution code (ZIP). Model files and the font are not included; official download addresses and SHA-256 hashes are recorded. Synthetic data and results are shared under CC0; original code under the MIT license.

Publisher: Codariva. Synthetic data generation, code, execution, and analysis were prepared with support from OpenAI Codex. Numerical totals and file integrity were verified through automated checks. No claim of human expert review is made.

Download the initial data together with the six-round timing addendum and evaluation check (ZIP). The initial ZIP is unchanged; the new package contains both experiments in separate folders.

Change log: 15 October 2026 — initial publication v1; timing addendum 1, six repetitions, error examples, and an independent numerical check were added the same day. The dataset was not expanded. Review was performed with Codex; no human expert review is claimed.

Planning a document pilot for your business

The scope of AI integration also includes data access and human approval. Use the pilot scenario and acceptance criteria guide to choose a suitable first task and prepare your project brief.