Extract Data From Receipts and Invoices
It is the twenty-eighth of the month, and a client has just sent a phone photo of a gas station receipt taken at an angle, in shadow, with a thumb in the corner. Somewhere in that blur is a total, a date, and a tax line you need for the books. You can read it in two seconds. The question worth asking is how you get a machine to read it as well as you do, and, more importantly, how you find out when it has not.
This article is for bookkeepers who handle receipts and vendor invoices for several small clients and want to stop retyping them. It compares three ways to extract data, then spends most of its time on the part that decides whether any of them is safe to use, which is validation, review, and measurement.
What you are actually extracting
Before choosing a tool, write down the fields you need. For most small-business bookkeeping the list is short.
- Vendor name and, if possible, a tax or registration number
- Invoice or receipt number
- Document date and, for invoices, the due date
- Currency
- Subtotal, tax, tip or shipping, and total
- Line items with description, quantity, unit price and amount
- Payment method or last four digits, for receipts
Header fields are the easy half. Line items are the hard half, because they live in tables whose shape changes from vendor to vendor. Many bookkeepers only need the header and the total, with the expense coded at the document level. If that is you, say so out loud, because it changes which method is worth paying for.
Three ways to read a document
Template OCR
Optical character recognition, usually shortened to OCR, turns pixels into text. Template OCR goes one step further. You tell the system that for vendor A the total sits in a box at a fixed position, and for vendor B it sits somewhere else. It then reads whatever text appears inside each box.
This works well when a small number of vendors send you the same format every month. Think of a utility bill, a rent invoice, or a supplier who exports from the same accounting package each time. It is cheap, fast, and predictable. It fails quietly when the vendor redesigns their invoice, when a long address pushes the table down a few lines, or when a scan arrives rotated. A template that was right for two years can be wrong on Tuesday, and it will not tell you.
Layout models
A layout model is a trained model that looks at both the text and where the text sits on the page. It has seen a very large number of invoices and receipts, so it learns that a number near the words "Total due" in the lower right is probably the amount owed, no matter where a given vendor puts it. You do not draw boxes. You send a document and get back fields, each with a confidence score.
Layout models handle new vendors without setup, which is the main reason to prefer them when your vendor list is long or changes often. Their weak spots are unusual documents such as handwritten receipts, multi-page invoices with continued tables, credit memos with negative amounts, and anything in a language or script the model saw less of. Bilingual documents are worth testing on purpose. A Puerto Rico vendor may print an English header and Spanish line items on the same page.
Language-model extraction
The newest option is to give the document, as an image or as OCR text, to a large language model and ask it to return the fields in a fixed structure. The model reads the whole document in context, so it copes with strange layouts, notes in the margin, and instructions such as "amount shown is in thousands".
The trade-offs are real. A language model can produce a confident, well-formatted answer that is simply wrong, for example a total that is the sum of line items when the printed total includes a deposit. It can also fill in a field that is not on the page. Results can vary between runs unless you pin the settings, and the cost per document is higher than the other two options. You will want to send documents containing client financial information only to a provider whose data handling terms you have read, and you should check what your own client agreements say about third parties. That is a contract question, so involve counsel if you are unsure.
A quick comparison
| Question | Template OCR | Layout model | Language model |
|---|---|---|---|
| Setup per new vendor | High | None | None |
| Handles odd layouts | Poorly | Reasonably | Well |
| Predictable output | Very | Mostly | Varies unless constrained |
| Per-document cost | Low | Low to moderate | Moderate to high |
| Typical failure | Silent shift after a redesign | Missed line items | Plausible but invented values |
| Best fit | A few stable vendors | Many vendors, standard invoices | Messy or mixed documents |
Most good setups combine them. A layout model does the first pass, a language model is called only when confidence is low, and a person looks at what remains.
Confidence scores are a signal, not a verdict
Every serious extraction service returns a confidence value per field. Treat it the way you treat a colleague saying "I think it's the fourteenth". Useful, but worth checking when the stakes are high.
Three habits keep confidence honest.
- Set thresholds per field, not per document. A date at 0.80 confidence is a different risk from a total at 0.80.
- Calibrate against your own data. A score of 0.95 does not mean it is right 95 times in 100 on your documents. You find out by counting, which is the evaluation set described below.
- Never auto-post on confidence alone. Confidence tells you where to look first. Validation rules tell you whether the numbers can possibly be correct.
Validation rules do most of the catching
A short list of arithmetic and consistency checks catches a surprising share of extraction errors, because errors rarely preserve arithmetic.
- Subtotal plus tax plus shipping equals total, within a cent or two of rounding.
- The sum of line amounts equals the subtotal.
- Quantity times unit price equals the line amount.
- The date is real, is not in the future, and is within a sensible window of today.
- The due date is on or after the invoice date.
- The currency matches the vendor's usual currency, or is flagged.
- The vendor matches a known vendor in the vendor master, or is routed to a person for creation.
- The invoice number plus vendor plus total has not been seen before, which catches duplicates.
- The tax rate implied by tax divided by subtotal is one of the rates that vendor normally charges.
When a check fails, do not try to repair the data automatically. Route the document to review and record which rule failed. Those failure reasons become your most useful reporting later.
A worked example
This is a hypothetical, with invented numbers, to show how the pieces fit.
Say a bookkeeper handles about 600 documents a month across five small clients. Before automation, each takes about two minutes to key in, which is 1,200 minutes, or 20 hours.
Suppose a layout model handles each document and the rules above run afterward. In the first month the bookkeeper sorts the results.
| Outcome | Documents | Share |
|---|---|---|
| All fields pass checks and confidence is high | 420 | 70 percent |
| Passed checks but one field had low confidence | 90 | 15 percent |
| Failed an arithmetic or duplicate check | 60 | 10 percent |
| Unreadable or unsupported (handwritten, torn) | 30 | 5 percent |
The 420 clean documents still get a light touch. The bookkeeper spot-checks a random 10 percent, which is 42 documents at perhaps 20 seconds each. The 90 low-confidence documents take about 45 seconds, and the 60 failures take about 90 seconds. The 30 unreadable ones are keyed by hand at two minutes.
The arithmetic is 42 times 20 seconds, plus 90 times 45 seconds, plus 60 times 90 seconds, plus 30 times 120 seconds. That is 840 plus 4,050 plus 5,400 plus 3,600, or 13,890 seconds, which is about 3.9 hours. Down from 20 hours, with a human still looking at everything that might be wrong. The savings are illustrative. Your split will differ, and the first month will be worse than the third while you tune thresholds and teach the vendor master about the vendors you actually have.
The more important number is the one the bookkeeper should record, which is how many of the 420 auto-passed documents contained an error found by the spot-check. If it is zero out of 42, you have good news but not proof. If it is three, you have just learned your threshold is too loose.
Where a person belongs in the loop
Human review should be designed, not improvised. A workable pattern has three lanes.
- Auto-accept. All required fields present, every validation rule passes, every field above its threshold, vendor known. A random sample is still reviewed.
- Quick review. One or two low-confidence fields. The reviewer sees the image beside the extracted values, with the doubtful field highlighted, and fixes it in seconds.
- Exception. A failed rule, an unknown vendor, a suspected duplicate, or an unreadable document. This goes to a named person with a reason attached.
Two details make review fast. Show the source image next to the fields so no one has to open another window, and capture every correction. Each fix is a labeled example you can feed back into your evaluation set.
Building an evaluation set you can trust
This is the step most small teams skip, and it is the step that lets you say "this works" instead of "this seems to work".
- Collect real documents. Pull 150 to 300 from the last few months across your clients. Include the ugly ones. A set made of clean PDFs will flatter any tool.
- Cover the range. Include several vendors with high volume, several with one document each, photos of receipts, scans, multi-page invoices, credit memos, and at least some bilingual items.
- Label by hand. For each document, record the correct value of every field you care about. Have a second person check a sample of the labels, because labeling mistakes make a good tool look bad.
- Freeze it. Keep this set separate and do not tune on it. If you adjust your rules while looking at the same documents you score on, your numbers will be optimistic.
- Score per field. For each field, compute the share extracted exactly right. For amounts, use exact match. For dates, normalize the format first. For vendor names, decide ahead of time whether "Acme Supply" and "ACME SUPPLY INC" count as the same.
- Score the system, not just the model. Count how many wrong values would have been posted after confidence thresholds and validation rules. That number is the one that matters to your clients.
- Rerun on a schedule. Run the set again whenever you change tools, prompts, thresholds, or a model version, and once a quarter regardless. Vendors change formats and providers update models without much notice.
An evaluation set of a couple hundred documents will not give you statistical precision. It will tell you which tool is clearly worse, which field is the weak point, and roughly what to expect. That is enough to make a decision.
Failure modes to plan for
- Transposed or dropped digits. A total of 1,284.50 read as 1,248.50 will still pass most formatting checks. Arithmetic against line items is your defense, and so is comparing to the bank feed amount at reconciliation.
- Wrong total chosen. Documents often show several amounts, such as subtotal, total, amount paid, and balance due. Make the rule explicit about which one you want.
- Locale formats. A date of 03/04/2026 is the third of April or the fourth of March depending on the sender. Decimal commas versus decimal points can change an amount by a factor of a thousand.
- Duplicates. The same invoice arrives by email, again as a reminder, and again as a photo. Duplicate detection by vendor, number and amount is not optional.
- Fabricated values. Language models sometimes return a field that is not on the page. Require the model to quote the text it used, and check that the quote actually appears in the OCR output.
- Privacy slips. Receipts can contain partial card numbers, names and addresses. Limit who can see raw images, set a retention period that matches your obligations, and keep the originals attached to the transaction for audit.
- Silent drift. Accuracy declines slowly as formats change. Your quarterly rerun is the alarm.
Questions to ask a vendor
- What is the accuracy per field on documents like mine, and can I test with my own set before I pay?
- Do I get field-level confidence and the position of each value on the page?
- Where are documents processed and stored, for how long, and are they used to train models?
- How do I export my data and my corrections if I leave?
- What happens to price when volume doubles?
- How are model updates announced, and can I pin a version?
What to do on Monday morning
- Write down the fields you truly need, and drop the ones you only wish you had.
- Pull 100 recent documents from at least three clients and put them in a folder.
- Label them in a spreadsheet, with a second person checking ten of them.
- Try two tools on the folder and score each field.
- Draft your validation rules from the list above and see how many of the known bad extractions they would have caught.
- Decide the three lanes and who owns the exception queue.
A note for Puerto Rico practices. Hurricane season is a good reason to keep the originals in a place that does not depend on one office or one laptop, and to keep the exception queue usable from a phone if power or connectivity is patchy for a few days. Documents in two languages are normal here, so make sure whatever you test includes them.
If you would like a second pair of eyes on your evaluation set or your validation rules, Ozcorp InfraOps builds and runs this kind of document pipeline for small teams, and I am happy to talk it through even if you decide to keep it in-house.
Osvaldo D. Rivera, PhD
Discuss your infrastructure requirements
Review the current service offer or schedule a conversation about the scope you need.
Stay in touch
Follow Ozcorp
Leave your details for occasional notes about new articles and Ozcorp updates.