Structured extraction from documents that have no fixed structure
A service that takes an arbitrary bank statement PDF and a user's known transaction list, and returns the transactions missing from their records.
Problem
Reconciliation is the step where bookkeeping usually breaks down: a business has its own record of transactions, the bank has the truth, and the gap between them has to be found and explained.
Automating it requires turning a statement PDF into structured transactions. That sounds like a solved problem and is not, because bank statements are a category of document with no shared format.
Constraints
- Layout varies per institution and sometimes per account type within the same institution. Column order, date formats, and how debits and credits are distinguished are all inconsistent.
- Some statements are digital text and some are scans, so text extraction alone is not sufficient.
- Transfers between a user's own accounts appear on both sides and must not be counted as missing.
- The operation is slow enough that a user needs to see progress rather than a spinner.
My role
Engineer on the extraction pipeline and the service that wraps it.
Architecture
A staged pipeline, where each stage validates before the next one runs: confirm the upload is actually a bank statement, detect the statement period, rasterize pages and extract transactions per page, then match the extracted set against the user's records.
Pages are rendered to images with PyMuPDF and passed to a vision model, which sidesteps the layout problem entirely — the model reads the page the way a person does rather than depending on the PDF's internal text ordering.
Progress streams back over server-sent events so each stage is visible while it runs.
The same pipeline is exposed both as a standalone service and through the reconciliation flow of the larger accounting platform.
Key decisions
Vision over text parsing
The alternative was per-bank parsers or a general PDF text extractor with heuristics for column detection. Per-bank parsers do not scale and break silently when a bank changes its template. Text extraction loses the spatial relationships that make a statement readable. Rendering to an image and using a vision model trades cost per page for not having to model layout at all, which is the right trade when the format space is effectively unbounded.
Validate the document before spending on it
The first stage simply asks whether the upload is a bank statement. It is cheap, and it prevents the expensive per-page extraction from running against a payslip or an invoice someone uploaded by mistake — and prevents the confident, entirely wrong output that would follow.
Rules for transfer matching, model for semantic matching
Transfers between a user's own accounts are identifiable from amounts, dates and direction. That is arithmetic, so it is written as rules with the model only as a fallback for genuinely ambiguous cases. Semantic matching is used where descriptions differ in wording but refer to the same transaction, which is the part that actually needs interpretation.
Stage-by-stage validation instead of one large call
Passing an entire statement to a model in one call is simpler and much harder to debug. Splitting into stages means a failure is attributable — period detection failing looks different from extraction failing — and each stage's output can be checked before the next consumes it.
Result
The service accepts statements from institutions it has never seen, extracts transactions, and reports what is missing from the user's records, with progress visible throughout.
Because format handling lives in the vision stage rather than in per-bank code, supporting a new institution generally requires no code change.
What I learned
- Vision models changed what is worth building for document work. A large amount of layout-parsing code written before them is now liability rather than asset.
- The instinct to send the whole document in one call should usually be resisted. Staged pipelines cost more calls and repay it in debuggability the first time something goes wrong.