How to Design a Document Processing Workflow
A box of scanned invoices, a shared inbox full of signed forms and a folder called "Misc 2" all have one thing in common. Nobody can say, with confidence, what has been dealt with. If an auditor, a client or a judge asked you to show that a given document arrived on a given day, was read by a particular person and was kept for the right length of time, how long would the answer take?
A document processing workflow is the answer to that question, built ahead of time. It is a defined path every document follows from arrival to disposal, with a record of each step. You can build it with paper trays and a clipboard, with a shared drive and some discipline, or with software that reads the documents for you. The design principles are the same in each case, and getting them right matters more than which tool you pick.
Think in stages, not in tools
Most workflows I see fall apart because they were designed around a product. Start with stages instead. Each stage has an input, a job, an output and an owner.
| Stage | Job | Output | Typical owner |
|---|---|---|---|
| Intake | Receive the document from any channel | A stored original with an ID and a timestamp | Front office or an automated mailbox |
| Identification | Decide what type of document it is | A document type label | Classifier or clerk |
| Extraction | Pull out the fields you need | Structured data | Reader software or clerk |
| Validation | Check the data against rules and sources | Pass, fail or needs review | Rules engine and reviewer |
| Routing | Send it to the person or system that acts on it | An assignment | Workflow rules |
| Action | Do the business task | A completed task | Process owner |
| Filing | Store the final version with metadata | A retained record | Records manager |
| Disposal | Destroy or transfer at end of retention | A disposal record | Records manager |
Notice that the original document never changes. Every later stage adds information next to it. That one rule, keep the original untouched and add metadata beside it, is what makes an audit possible later.
Intake, where chaos begins
Documents arrive through more doors than you think. Email attachments, a client portal, a scanner, a phone photo sent by text message, a fax line that nobody has turned off, and paper brought to the counter. List every door before you design anything, and then close the unofficial ones one at a time.
Good intake does four things.
- Assigns a unique ID the moment the document arrives.
- Records the channel, the sender and the time received.
- Stores the original in a protected location that most people can read but few can edit or delete.
- Rejects or flags files that cannot be read, such as password-protected PDFs, corrupted files and empty scans.
Give clients one preferred door and tell them plainly, in English and Spanish if you serve both, what format and quality you need. A phone photo taken at an angle in poor light will cost you more time at every later stage than a minute of guidance up front.
Identification and extraction, and the honest limits of reading machines
Once a document is in, you have to know what it is. A person can do this in two seconds. Software can do it too, and the question is how much you trust it.
There are three broad ways to read a document.
- Structured data. If a document arrives as a form on a web page or a spreadsheet with fixed columns, there is nothing to read. The fields already exist. Prefer this wherever you have influence over the sender.
- OCR, short for optical character recognition. Software converts the picture of text into actual text. Clean printed pages work well. Handwriting, stamps, skewed scans and faint copies do not.
- Language-model extraction. A model reads the text and fills in named fields, such as date, parties and amount. It copes with varied layouts but can produce a confident wrong answer, so it needs checks that do not depend on the model agreeing with itself.
Treat every automated reading as a proposal, not a fact. That attitude leads straight to the next concept.
Confidence thresholds, the dial that controls your workload
Most extraction tools report a confidence score for each field, usually a number between 0 and 1. A threshold is the line above which you let the result through without a person looking, and below which it goes to review.
Setting that line is a trade-off between two kinds of cost.
- Set it too low and wrong data flows through. A wrong amount on a payment is expensive.
- Set it too high and reviewers drown in documents that were actually fine.
Do not copy a number from a vendor. Measure your own. Take 100 to 200 real documents, have a person record the correct values, run them through the tool, and compare. Then look at accuracy by field and by document type. You will often find that dates are reliable and handwritten amounts are not, which means you can set different thresholds for different fields.
A hypothetical illustration, with invented numbers. Suppose you test 200 invoices and find that for the total amount field, results scored at 0.95 or above were correct 199 times out of 200, while results scored between 0.80 and 0.95 were correct only 70 percent of the time. You might auto-accept the first group, send the second group to a person, and send anything under 0.80 to full manual entry. Re-run the test every few months, and again whenever the tool, the vendor mix or the scanner changes.
Confidence is not the only gate. Add business rules that catch mistakes a model will not. Line items should add to the total. A date should not be in the future. A client ID should exist in your system. A signature page should have a signature. These checks are cheap and often find more errors than the confidence score does.
Queues and the people who work them
A queue is simply a list of documents waiting for a person. Queues are where a workflow either runs smoothly or quietly rots, so design them with care.
- Separate queues by skill and urgency, not by arrival time alone. A document needing a licensed professional's eyes should not sit behind routine filing.
- Give every item an owner. "The team" is not an owner. If everyone can pick it up, nobody does.
- Set a service target for each queue, such as "reviewed within one business day", and show the age of the oldest item where the team can see it.
- Plan for absence. In a small office, one person on leave or a hurricane warning that closes the building can stop everything. Define a backup reviewer and a way to work remotely with the same access controls.
- Limit what reviewers can change. They should be able to correct extracted values, but the change should be logged, and the original value should stay visible.
Exceptions are the real workflow
The happy path takes care of itself. What defines a good design is what happens to the rest. Write down the exceptions before launch and give each one a named destination.
| Exception | What to do |
|---|---|
| Unreadable or blank file | Return to sender with a specific request, and log the request |
| Unknown document type | Send to a person to classify, then add the new type to the rules |
| Duplicate of an existing document | Link to the original, do not process again, and note it |
| Missing page or signature | Hold, notify the sender, and set a reminder date |
| Conflict between document and system record | Route to the record owner for a decision |
| Contains something sensitive that was not expected | Restrict access immediately and notify the person responsible for privacy |
Duplicate detection deserves a word. A file hash catches an exact copy. It will not catch the same invoice scanned twice or emailed again with a slightly different PDF. For those, compare business keys such as sender, document number, date and amount.
Review the exception queue monthly. A growing pile of one exception type is telling you to change the front of the process, for example by adding a required field to the client request form.
Retention and disposal, the part everyone forgets
Keeping everything forever feels safe and is not. Old records raise storage costs, widen the damage of any breach, and can be demanded in legal disputes long after you needed them. Throwing things away too early is worse.
Retention is a rule that says how long a type of document must be kept, and what triggers the clock. The trigger matters. "Seven years" means little until you say seven years from what, such as the date of the document, the close of the engagement or the end of the contract.
The required periods depend on your industry, the type of document and the jurisdiction, in Puerto Rico and in the US at the federal and state level. I am not offering legal or tax advice, and you should confirm every period with your own counsel, accountant or regulator. What I can say is how to structure the answer.
- Build a retention schedule listing each document type, its retention period, its trigger, its legal or business reason and who approved it.
- Store that schedule where the workflow can read it, so each document is tagged with a disposal date at filing time.
- Add a legal hold, a flag that suspends disposal when litigation or an investigation is possible. Someone must own the right to place and release it.
- Dispose on a schedule, record what was disposed of and when, and make sure the copies in backups and email are covered. Destruction of paper should be done by a method you can document.
Audit, or how to prove it happened
An audit trail answers who did what, to which document, and when. Capture it as a side effect of the work so nobody has to remember to do it.
At minimum, record these events.
- Receipt, with channel and time.
- Each classification and extraction, noting whether a person or software did it.
- Each change to a value, with the old value, the new value and the user.
- Each view of a sensitive document, if your system supports it.
- Each approval, rejection and routing decision.
- Each disposal and each legal hold.
Make the log append-only, meaning entries can be added but not edited, and keep it for at least as long as the records it describes. Test it by picking a random document and reconstructing its whole life in five minutes. If you cannot, the log is not yet doing its job.
Access control belongs here too. Use roles rather than individual permissions, give each role the minimum it needs, and review who holds which role on a schedule. Confidential material such as health, financial or personnel records often needs a separate, tighter queue.
A small worked example
This hypothetical walk-through uses invented numbers. A 6-person accounting office receives roughly 400 client documents a month by email and portal upload.
- Intake assigns IDs and stores originals. About 20 files a month are unreadable, and the office sends each sender a standard request.
- The classifier labels documents as W-2, bank statement, invoice, or other. After a trial on 150 past documents, the office finds that it labels 90 percent correctly and sends the rest to a clerk.
- Extraction reads key fields. Amounts below a 0.95 confidence line go to a reviewer, which turns out to be about 15 percent of documents, or 60 a month.
- A reviewer handles those 60 in a queue with a one-day target. A backup reviewer is named for vacation weeks.
- Filed documents are tagged with client, year and a retention date from the schedule.
- Each month the manager reads the exception report and a sample of 10 audit trails.
The office did not eliminate human work. It moved human attention from every document to the ones that deserved it, and it gained a record it did not have before.
What to do on Monday morning
- List every channel through which documents enter your office.
- Write down your document types, then pick the three that cause the most trouble.
- For those three, draw the stages from the table above and name an owner for each.
- Pull 50 recent documents and note which ones would have been exceptions, and why.
- Draft a one-page retention schedule and take it to your counsel or accountant for review.
- Decide where the audit log will live, even if it starts as a simple spreadsheet.
If you want a second pair of eyes on a design like this, Ozcorp InfraOps works with small teams on exactly these workflows. InfraOps Base starts at $500 per month with vendor and license charges billed separately, and a conversation costs nothing.
Discuss your infrastructure requirements
Review the current service offer or schedule a conversation about the scope you need.
Stay in touch
Follow Ozcorp
Leave your details for occasional notes about new articles and Ozcorp updates.