Four classes, four denominators, deliberately never averaged into one number.
Only document_extraction measures the parser;
source_of_record measures a vendor backfill and cannot
substitute for it. Real and synthetic corpora are separate tiers.
Coverage matrix
Computed over runnable extraction cases only — a catalogued document that cannot be
driven through the pipeline contributes no evidence about the parser and does not pad
the matrix. Adding a document should close a named cell, not increment a total.
Named gaps
Catalogued cases that cannot currently be driven through the pipeline, each with the
reason stated. Visible gaps beat hidden ones.
Run an eval
Each run mints a unique pipeline_release tag. This is not
cosmetic: pipeline_jobs is unique on
(document_id, pipeline_release) and its upsert does not reset
job state, so reusing a tag would return an already-finished job and grade the previous
run's metrics while reporting success. A reingest drives real documents through the real
pipeline serially and can take hours; rescore is fast but can
only detect a scoring change, never a parsing regression.
Add a candidate document
An uploaded document is a candidate, not an eval case. It is registered
as unlabeled, excluded from runs and from every accuracy
denominator, until ground-truth labels are attached and it is promoted to
golden. A document with no expected values adds coverage on
paper and zero measurement in practice.