Two different jobs hiding inside "document review"
"Review this pile of documents" sounds like one task. It is actually two, and they behave nothing alike.
The first is extraction: pull specific fields out of every document — a policy number, a contract's effective date, an applicant's declared income, a supplier's insurance expiry. This is a well-defined task with a checkable answer. Either the extracted date matches what's printed on the page or it doesn't, and a person can confirm that in seconds by glancing at the source. It's tedious at volume, but it isn't ambiguous.
The second is assessment: does this clause sit outside what the firm normally accepts, does this claim's account of events look consistent with the supporting evidence, does this application raise something an eligibility rule doesn't explicitly cover. There's no single correct answer sitting on the page waiting to be copied out — assessment depends on judgment, on what "normal" looks like for this business, on context a checklist can't fully capture in advance.
That difference matters because it's where the value and the risk both concentrate. Extraction saves time; get a field wrong and it's usually caught the moment someone checks it against the source. Assessment is where a genuinely bad outcome hides — a missed exclusion clause, a claim that should have been flagged and wasn't, an application that slipped past a rule nobody thought to write down explicitly. A useful system has to treat these as two different jobs with two different standards, not one blended "review" step that quietly borrows extraction's confidence for assessment's harder calls.
The manual way vs. the automated first pass
Manual review usually means one person opening one document at a time: reading it in full or scanning for known terms, typing the fields they find into a spreadsheet or case system by hand, and separately forming a judgment about anything that looks off — all under whatever time pressure the pile of remaining files creates. What counts as "worth flagging" quietly shifts depending on who's reading, how many files are left, and how the last few similar documents turned out.
An automated first pass does the same two jobs, structured so a person reviews results instead of raw pages. The system opens each document, pulls the checklist fields with a citation attached to each one, and produces a separate assessment layer that surfaces anything worth a second look — again with the exact passage attached, not just a verdict. A reviewer's day starts with a stack already sorted into "fields captured, nothing flagged" and "here's specifically what needs your eyes," rather than a blank read of every file in the order it arrived.
| Manual review | Automated first pass | |
|---|---|---|
| Extraction | Typed by hand from the page, one document at a time | Pulled automatically, every field tied to its source passage |
| Assessment | Formed by whoever is reading, under time pressure | Flagged for a person, with the passage that triggered the flag |
| Consistency | Varies by reviewer, by day, by how many files are left | The same checklist and criteria applied to every document |
| Verification | Requires re-reading to double-check a field | A cited passage confirms or corrects a field in seconds |
| Final call | Made by the person reading | Still made by the person reading |
That last row is the point worth sitting with. Automating the reading doesn't automate the decision — it changes what the person spends their limited attention on, from re-typing fields off a page to weighing the handful of things that actually needed a human judgment call.
Documents that don't cooperate: scans, bad formatting, and versions that drift
Real document stacks are rarely clean, consistent PDFs. A claims file might include a faxed form scanned a second time by whoever received it. A supplier's contract template looks nothing like the one your own legal team drafts, and no two suppliers format theirs the same way. A grant application submitted this year uses a different layout than the one submitted last cycle for the same programme.
The first hurdle is simply reading what's on the page. A scanned or photographed document needs optical character recognition — software that turns an image of text into text a system can actually search and extract from — before any of the extraction or assessment work can start, and a poor-quality scan degrades that step the way it would degrade a person's ability to read it. The second hurdle is structure: tables that don't line up the same way twice, handwritten annotations in a margin, a form field that moved between one version of a template and the next. The third is version drift — a contract gets amended, a claim file gets supplemented with new evidence weeks later, an application gets resubmitted with corrections — and the system has to know it's reading the current version rather than silently blending a superseded clause into a current answer, which is a worse failure than simply being slow.
None of this gets solved by pretending it doesn't happen. It gets handled by building for it from the start, and by being honest about where confidence genuinely drops — which is exactly where the next section picks up.
How a build actually works
Every answer comes with its passage, not just its answer
A field or a flag with no source attached asks a reviewer to trust it blind, which is the opposite of what fast, reliable review requires. A build worth using shows the exact passage — the sentence, the clause, the line on the form — that produced each extracted field and each assessment flag, so a person can confirm it against the original in the time it takes to glance at two things side by side, rather than re-reading the whole document to check one line. This is the difference between a tool a reviewer trusts and one they quietly stop using: an answer with nothing behind it invites doubt, and a bare assertion is never good enough on its own, no matter how confident it sounds.
This is also where extraction from PDFs and other source formats has to hold up under pressure — a field pulled from automated data extraction is only as useful as the citation that lets someone verify it landed in the right place.
The low-confidence pile goes to a person, not a guess
Every extracted field and every assessment flag carries a confidence level, not just a value. A clean, typed document with an unambiguous field produces high confidence. A badly scanned page, a term with no clean match to anything in the checklist, or a clause structured in a way the system hasn't seen before produces low confidence — and low confidence doesn't get smoothed over with a best guess presented as fact. It gets routed to a person, marked as needing review, exactly the same way a claims handler would set aside a file they weren't sure about rather than approve it on a hunch. That routing decision is what human-in-the-loop design actually means in practice: build the system to hand off the cases it isn't sure about, instead of quietly guessing and hoping nobody checks.
Whatever industry runs this — a claims team, a procurement desk, a compliance function, or a firm managing legal matters — the output is a first pass for a qualified person to check, never a decision the system makes on its own. That holds regardless of what the documents are or what's riding on getting them right.
What it connects to
A document review system is only useful if it reaches the documents you actually receive and lands its output where your team actually works. Typically that means:
- Wherever documents arrive — an inbox, a claims portal, a supplier onboarding form, an application intake system — read as they come in rather than batched for a manual sweep later.
- Your case or matter management system, so extracted fields and flags land as structured records rather than a separate spreadsheet someone has to reconcile by hand.
- Whatever holds the checklist and the rules, so extraction fields and assessment criteria reflect what your team actually requires, not a generic template.
- Your document storage, so the source file stays one click away from every cited passage, and a superseded version doesn't quietly outrank the current one.
None of this requires replacing what your team already runs. Where a system exposes an API, a build connects to it directly; where an older platform doesn't, there's usually a workable route in through an export or a webhook it already supports. When the stack is entirely contracts rather than a mix of document types, a narrower contract analysis build is often the better starting point, and either shape sits under the same knowledge systems work more broadly.