Golden Eval Corpus — Reference

1. What this is

A curated set of documents with known-correct expected values, plus the machinery to score the V5 pipeline against them on demand and keep the results as a time series.

Before this existed, the corpus was four unrelated directories on disk with no index, and every scoring run wrote to a file under .audit/. Two questions had no answer short of reading directories and interpreting them by hand:

• What is in the corpus, and what does it cover?
• Did accuracy move, and which specific assertions changed?

Both are now queries. The catalog lives in eval_corpus_documents / eval_corpus_labels; scoring history lives in eval_runs / eval_run_documents / eval_results.

The distinction that everything else follows from

2. The four eval classes

The corpus proves four different things with four different denominators. They are never averaged into a single number, because a blended percentage across them measures nothing while sounding authoritative. eval_class lives on the document and every aggregate is keyed by it.

Real vs synthetic is a separate axis

corpus_kind is real or synthetic and the two are reported as separate tiers. Synthetic supply is effectively unbounded — the Blackwell generator emits the document and its expected facts together — but it cannot demonstrate real-world accuracy. Merging it into the real-document headline would make the corpus look stronger and the number weaker.

Label evidence grade

label_authority_class records how much the ground truth itself is worth. This is about the labels, not the document.

3. Invariants

These are enforced by schema constraints and code, not by convention. Breaking one produces a dashboard that lies, which is worse than no dashboard.

1. Observer only — never modify the pipeline

The eval writes exclusively to eval_* tables. It never writes to, alters or migrates documents_v5, pipeline_jobs, extracted_metrics, document_canonical, document_chunks_v5 or pipeline_outbox. It drives the pipeline only through the pipeline's own pre-existing HTTP surface and reads results back with plain SELECTs. There are deliberately no foreign keys from eval_* into pipeline tables — so eval history survives pipeline row cleanup, and this feature can never block a pipeline migration. scripts/apply-eval-corpus-migration.js verifies both properties on every apply and fails loudly.

2. Every ratio ships with its denominator

Per-axis samples here are small. A percentage without its sample size is not a measurement. eval_run_documents.denom is stored explicitly and the UI renders n= beside every figure.

3. Null means "no evidence", never zero

When nothing was asserted, pass_rate is NULL and the UI renders "no evidence". A 0% that actually means "we measured nothing" is the single most misleading thing an accuracy dashboard can show. Note the corollary in gating: a gated statement with coverage = null is a gate failure, so an empty corpus surfaces the gap instead of being rewarded.

4. Unjudged is not a failure

The label set is a sample, not a census. An extracted observation outside label coverage is counted in n_unjudged and reported, never scored as a false positive. Treating uncovered observations as false positives would understate precision by construction.

5. Exclusions are explicit and reasoned

eval_corpus_labels.is_scored = false requires exclusion_reason (CHECK-enforced). Placeholder zeros and skip-flagged stubs are excluded from both numerator and denominator, and say why. Same for runnable = false requiring not_runnable_reason.

6. System output never becomes a label

Ground truth is authored from the document, not from what the pipeline produced. The Harbor dictionary models this properly: it declares authority: fixture document text, an explicit never_consulted: list naming the tables it refused to read, and per-fact confirmed_by. If label pre-fill from extraction is ever added, those labels must carry a distinct authority class and stay out of every denominator until a human confirms them.

7. No blended score

There is no column anywhere that holds an average across eval classes, across real and synthetic, or across the six Harbor fields.

4. Architecture

Admin console  public/internal/eval-corpus.html
       │  fetch + x-admin-key
       ▼
Controller     src/admin/eval-corpus.controller.ts   @Controller('admin/eval-corpus')
       │                                             @UseGuards(PlatformAdminGuard)
       ▼
Service        src/admin/eval-corpus.service.ts      EvalCorpusService
       │
       ├── catalog / coverage reads ──────────► eval_corpus_documents, eval_corpus_labels
       │
       ├── startRun() ─► mints eval_runs row (state=queued), returns runId, detaches
       │                     │
       │                     ▼  executeRun()  (background)
       │     ┌───────────────┴────────────────────────────┐
       │     │ public_equity leg                          │ private_equity leg
       │     │ spawn scripts/v5-accuracy-eval.js          │ spawn scripts/run-harbor-
       │     │   env: PIPELINE_RELEASE, GT_INCLUDE,       │   accuracy-eval.js --json
       │     │        SKIP_UPLOAD, FUNDLENS_BASE_URL      │ (read-only, no re-ingest)
       │     │   └─► POST /api/v5/dev/upload              │
       │     │        x-pipeline-release: <run tag>       │
       │     │        polls pipeline_jobs.state           │
       │     │   └─► writes .audit/v5-accuracy/<slug>.json│
       │     └───────────────┬────────────────────────────┘
       │                     ▼
       │            ingest artifacts ──► eval_run_documents, eval_results
       │                     │
       │                     ▼
       │            observePipeline()  SELECT-only:
       │              pipeline_jobs.parser_name / state
       │              document_layout.parser_name
       │              llm_invocations  → cost_usd, call count
       │              pipeline_events  → stage wall-clock
       │                     │
       │                     ▼
       │            buildSummary() ──► eval_runs.summary, gate, cost, duration
       ▼
Backfill       scripts/backfill-eval-corpus.js  (CLI, idempotent)
                 indexes the 5 on-disk sources into the catalog

Why the service spawns the existing scripts

scripts/v5-accuracy-eval.js holds roughly 500 lines of comparator logic that is almost entirely accumulated edge cases, each one a real defect that was found and fixed: balance-sheet snapshot period aliasing (gated so a 9M revenue can never satisfy an FY pin), sign-wildcard absolute-value comparison for capex printed in parentheses, an is_primary tie-break added because a non-GAAP "Adjusted FCF" was shadowing the primary statutory FCF, a ~35-entry bidirectional statement-role alias map, and registry-derived canonical-id alias merging.

Reimplementing that in TypeScript would create a second implementation of one rule. This repo's leading defect class is exactly that — the Harbor eval header records a case where the duplicate had a bug production did not. So the comparator stays where it is, proven, and the service is orchestration and persistence around it.

Accepted tradeoff: an out-of-process invocation is coarser than an in-process call and cannot stream progress per metric. It does report per-document progress, which is the granularity the console needs.

5. Database schema

Migration: prisma/migrations/20260821_eval_corpus.sql. Apply with node scripts/apply-eval-corpus-migration.js. The COMMENT ON statements in that file are the authoritative reference — this page summarises them.

eval_corpus_documents — the case registry

One row per corpus artifact. "Document" is loose: a product_logic row has no file, and carries runnable = false with a reason.

eval_corpus_labels — the expectations

Deliberately not unified into one flat shape. Three label vocabularies already existed and each carries load-bearing distinctions a generic {expected_value, tolerance} would destroy. The native record is preserved verbatim in expected (JSONB); the promoted columns exist only so common questions are indexable.

Notable columns: axis_pin_count (0–8) makes "is our per-axis coverage real or aspirational" a query rather than a guess; is_scored + exclusion_reason implement invariant 5.

eval_runs — one row per triggered run

Modelled on strategy_dna_evaluations (counts + JSONB detail + prior_*_id delta), the closest existing precedent in this repo.

eval_run_documents — per (run, document)

Exists because the facts that diagnose a run are not derivable from aggregates: which parser ran, where the job stopped, how long it took, what it cost. UNIQUE (run_id, corpus_document_id).

Key columns: parser_name (see §9 — the harness never recorded this), final_job_state, eval_tenant_id / v5_document_id / v5_job_id (observed, not FKs), n_pass / n_fail / n_missing / n_skipped / n_unjudged, pass_rate + denom, and canonical / chunks / per_axis JSONB observations.

eval_results — the atomic scored assertion

One row per (run, label). This is the grain that makes a regression diagnosable at assertion level rather than as a moved percentage.

Also carries expected_value, actual_value, tolerance, delta_pct, matched_metric, grounded, evidence_type, axis_status JSONB (the 8 axes) and field_status JSONB (the 6 Harbor fields).

6. API reference

Base: /api/admin/eval-corpus. Every endpoint requires x-admin-key matching PLATFORM_ADMIN_KEY (or PLATFORM_ADMIN_KEY_SECONDARY). Failures return a bare 401 with no body, so the endpoint's existence isn't revealed. There is no tenant or user context — the admin key is the authorization.

POST /runs

{
  "scope":     "smoke" | "regression" | "full" | "custom",   // default "full"
  "mode":      "reingest" | "rescore",                        // default "reingest"
  "partition": "public_equity" | "private_equity",            // optional, both if omitted
  "slugs":     ["crh-fy2024", "meli-fy2024"],                 // optional; overrides scope
  "label":     "pre-release check",                           // optional, ≤200 chars
  "triggeredBy": "ajay",                                      // optional
  "targetRelease": "v5-f26-f27a"      // rescore ONLY; 400 if sent with reingest
}

Returns 202, not 200: a full reingest drives every corpus document through the real pipeline serially and is measured in hours. Poll GET /runs/:runId.

Scope semantics: smoke = 1 document, regression = 2, both ordered by source_bytes ascending. Cost is dominated by page count, and a smoke tier that takes an hour will not get run. Selection is always restricted to eval_class='document_extraction' AND runnable AND label_status='golden'.

POST /:slug/labels

{
  "labelKind": "metric_axes",          // default; or pe_fact | qa_check | line_item_structure
  "labels": [
    { "canonical_id": "revenue", "statement": "income_statement",
      "period": "FY2024", "expected_value": 34351880400, "tolerance": 0.05,
      "expected_role": "revenue", "expected_currency": "USD", "expected_sign": "positive" }
  ],
  "promote": true,                     // default true → label_status becomes 'golden'
  "authority": "10-K page 62, PDF-verified 2026-08-21",
  "authorityClass": "primary_document" // 'unpopulated' blocks promotion
}

Validation is strict and per-label: a metric_axes label without a finite expected_value is rejected and reported in rejected[] rather than silently accepted. A bad label is worse than a missing one, because it produces a confident wrong verdict that then gets chased as a product defect. If every label is rejected the whole request 400s.

POST /upload

Limits: 150 MB, one file, extension allow-list (pdf docx doc xlsx xlsm xls csv pptx html htm md txt). Requires partition. Optional metadata: company, ticker, period, filingType, accountingStandard, currency, language, jurisdiction, docGenre, corpusKind, notes, uploadedBy.

Deduped on content hash — identical bytes return the existing case with duplicate: true rather than creating a second row that would double-count in every coverage figure. Registers as unlabeled, which means excluded from runs and from every denominator until labelled and promoted.

7. Runbook

Environment

First-time setup

# 1. create the eval tables (idempotent; verifies observer-only and fails on drift)
node scripts/apply-eval-corpus-migration.js

# 2. index the on-disk corpus into the catalog
node scripts/backfill-eval-corpus.js --dry-run   # report only, no writes
node scripts/backfill-eval-corpus.js             # apply + print coverage matrix

# 3. open the dashboard
open http://localhost:3000/internal/eval-corpus.html

The backfill is idempotent and safe to re-run — re-run it whenever ground-truth files change on disk. It upserts on slug and on the label identity expression, and runs inside a single transaction so a parse failure anywhere leaves the catalog untouched rather than partially updated (a partial catalog silently understates coverage).

Running an eval

From the console: Runs tab → pick scope and mode → Start run. The page polls every 15 s and refreshes the catalog when the run finishes.

From a terminal:

KEY=$(grep -m1 '^PLATFORM_ADMIN_KEY=' .env | cut -d= -f2- | tr -d '"')

# fast: grade already-stored rows (seconds, no pipeline cost)
curl -s -X POST -H "x-admin-key: $KEY" -H 'Content-Type: application/json' \
  -d '{"scope":"custom","mode":"rescore","slugs":["crh-fy2024","meli-fy2024","recruit-fy2026q3"]}' \
  http://localhost:3000/api/admin/eval-corpus/runs

# full: drive documents through V5 (hours, real Bedrock cost)
curl -s -X POST -H "x-admin-key: $KEY" -H 'Content-Type: application/json' \
  -d '{"scope":"full","mode":"reingest"}' \
  http://localhost:3000/api/admin/eval-corpus/runs

# poll
curl -s -H "x-admin-key: $KEY" \
  http://localhost:3000/api/admin/eval-corpus/runs/<runId> | jq '.run.summary'

Choosing a mode

Runs are serialized — starting one while another is active returns 400. The harness writes per-document artifacts to fixed paths under .audit/v5-accuracy/, so concurrent runs would overwrite each other's evidence and both would report whichever finished last.

Adding a document to the corpus

Two routes. Prefer the first for anything permanent.

A. On disk (durable, version-controlled — the system of record)

1. Put the artifact under COMPANY-FILINGS/.
2. Author benchmarks/ground-truth/<slug>.yaml following benchmarks/ground-truth/GT_FORMAT_GUIDE.md. Set source_file to the repo-relative path and data_source to an honest provenance string — if the values came from a vendor, say so.
3. Run node scripts/backfill-eval-corpus.js.
4. Verify it appears with runnable=true, then run a custom reingest over just that slug.

B. Via the console (curation convenience)

1. Add document tab → upload → registers as unlabeled.
2. POST /:slug/labels with expected values and an authority class → promotes to golden.
3. Run a custom eval over that slug.

Interpreting a result

• Read the per-class scoreboards, not a single number. Only document_extraction speaks to the parser.
• Check n= on every percentage. Small samples move a lot.
• Check parser_name per document. A pdf-ocr document that stalled at parsing_submitted did not fail — it could not complete in that environment (see §9).
• Read regressions[] before the headline. It names the assertions that were passing and now are not, with expected vs actual.
• label_authority_class on the contributing documents caps how strong the claim can be. A corpus of vendor_api labels scoring 100% proves agreement with a vendor, nothing more.

8. Design decisions and why

9. Traps and gotchas

10. Current state (as at 2026-08-21)

Catalog: 56 documents, 848 labels.

Verified reference run

Rescore over the 4 runnable public documents, graded against release v5-f26-f27a. Gate fail (income statement 88.5% against a 90% threshold). CRH's 94.7% reconciles exactly with running the harness standalone, which is the evidence that ingest is faithful rather than approximately right.

Open items

• 3 extraction defects found by that run, unfixed by design (measure first): Recruit revenue expected ¥2,736,780,000,000 extracted 0; gross_profit expected ¥1,611,128,000,000 extracted 0; basic_eps expected 5.06 extracted 4.21.
• J-GAAP has no runnable ground truth despite a shipped EDINET parser and J-GAAP mapper. Recruit is a Japanese issuer but reports IFRS.
• bmw-fy2024 cannot ingest. Fixing it would take primary-document parser evidence from 2 documents to 3.
• CRH labels are vendor-derived. Re-sourcing them from the 10-K PDF would make it a true primary-document case.
• 3 placeholder GT stubs (lvmh 16/16 skip-flagged, siemens 13/13, lund 4/5) marked corpus_eligible: false. Real ground truth would add three IFRS documents.
• 14 structure fixtures are not runnable — they assert line-item order for filings absent from the repo. Adding those filings would activate 421 assertions.
• Public-equity label supply could scale on XBRL. SEC/ESEF/EDINET facts are issuer-tagged and independent of our parser — a third-party label source at near-zero marginal cost. Subject to the granularity-match rule in .kiro/steering/source-precedence.md: label at matching disclosure basis or not at all.

11. Extending it

Add a new corpus source directory

Write a collect*() function in scripts/backfill-eval-corpus.js returning { doc, labels[] } records, and register it in the sources array in main(). Everything downstream (coverage matrix, UI, runs) is driven off the catalog and needs no change.

Add a new label vocabulary

Extend the eval_corpus_labels_kind_chk CHECK and eval_results_verdict_chk if new verdicts are needed. Keep the native record in expected verbatim; promote only the columns you need to index. Resist unifying into an existing kind — the vocabularies are distinct because the distinctions matter.

Add a new eval leg (a different scorer)

Follow the Harbor pattern in EvalCorpusService.runHarborEval(): spawn the scorer, parse its output, write one eval_run_documents row per corpus document and one eval_results row per assertion. Report its numbers beside the others — do not merge them into an existing scoreboard.

Persist eval runs from CI

Call POST /runs with trigger_source left default, or insert directly with trigger_source='cli'|'scheduled' (both already allowed by the CHECK). Gate thresholds in buildSummary() mirror scripts/lib/eval-aggregator.js (IS 0.90 / BS 0.85 / CF 0.80) so the console and CI cannot disagree about whether a run passed. If you change one, change both.

Make uploads durable

Store to S3 under an eval-corpus/ prefix and add a materialise-to-disk step before the harness runs, or require corpus artifacts to be committed. Do not simply switch to S3 without the materialise step — the harness reads from the filesystem.

12. File map

13. Change history and deployment

Commits

Unrelated work also present on this branch and not part of this feature: a230477 (JWT verification), a94bf2c (cost telemetry), 0320a64 (EDINET/DART metric storage).

Branched from feat/pe-uiux-phase4. Deliberately a separate branch: the eval corpus is unrelated to the PE UI/UX work in flight on that branch, and mixing them would make either one hard to review or revert independently.

Deploying

Order matters — the schema must exist before the code that queries it starts serving.

# 1. Schema. Idempotent, and self-verifying: it diffs pipeline table column
#    counts before/after and fails if any eval_* table gained an FK into a
#    pipeline table. Safe to re-run.
node scripts/apply-eval-corpus-migration.js

# 2. Deploy the application (the new controller + service).
npm run build
#    then the usual path: scripts/deploy/deploy-infrastructure.sh / ECS task update

# 3. Populate the catalog. Until this runs the dashboard renders an empty
#    corpus, which is correct but not useful.
node scripts/backfill-eval-corpus.js

# 4. Smoke check
curl -s -H "x-admin-key: $PLATFORM_ADMIN_KEY" \
  https://<host>/api/admin/eval-corpus/coverage | jq '.coverage.byClass'

Environment prerequisites on the target: DATABASE_URL, PLATFORM_ADMIN_KEY, and — only if you intend to run reingest evals there — V5_DEV_UPLOAD_ENABLED=true, since the harness drives /api/v5/dev/upload and that endpoint defaults off in production. A rescore needs neither the dev upload route nor Bedrock, so it works on a locked-down host.

Rollback

Reverting the application commit is sufficient and safe. Because the feature is an observer, the eval_* tables are inert once nothing queries them: no pipeline code reads them, no FK points into them, and no trigger fires from them. Leaving the schema in place preserves the measurement history for the next attempt.

git revert 9014a60        # or drop the branch before merge

# Only if you genuinely want the history gone. Destructive and NOT required
# for a rollback — every eval run ever recorded is in these tables.
# DROP TABLE eval_results, eval_run_documents, eval_runs,
#            eval_corpus_labels, eval_corpus_documents;

Known state at commit time

• Applied and verified against the live database used by this workspace: 5 tables created, pipeline table shapes unchanged, no FK leakage.
• Catalog populated: 56 documents, 848 labels.
• One verification run recorded (rescore, 4 public documents, graded release v5-f26-f27a) — see §10.
• Not yet deployed anywhere. Local only.
• Not yet pushed. The branch exists locally; push and open a PR when ready.

14. Growing eval depth (not document count)

The corpus has a depth problem, not a breadth problem. A 10-K presents two or three comparative years across three statements, 60–100 line items, and a segment note. We grade one period and ~15 headline metrics per document. Most of every document is unmeasured.

Run node scripts/eval-label-gap-report.js --release <release> to get this measured per document instead of estimated.

Measured headroom (public equity, 4 runnable documents)

Corpus: 66 labels today → 379 available (5.7×) from the same documents, and roughly 3,000 assertions once the eight axes are counted per label.

The four levers, cheapest first

1. Comparative periods. The single biggest and cheapest lever. The comparative column sits beside the one the author already read, and the author already knows the concept and statement. It also tests a real defect class that a single-period corpus cannot see at all: column selection. A parser that grabs the prior-year column is within tolerance of nothing and currently invisible. No schema or harness change needed — eval_corpus_labels is unique on (document, kind, key, period) and the harness resolves period per metric via expected_period_normalized, so extra periods work today.
2. Axis completion. A label pinning all eight axes yields eight assertions instead of one, with no additional document reading beyond what authoring the value already required. Currently 61 of 66 runnable labels are fully pinned, so this lever is nearly exhausted on the public side — which is why it ranks second rather than first.
3. Derived / footing relations. Free — composed from labels that already exist. Assert the arithmetic, not just the values: gross profit foots from revenue and COGS; assets equal liabilities plus equity; free cash flow equals operating cash flow less capex; net income on the income statement ties to the top of the cash flow statement. This catches a defect class per-value checks structurally cannot: a set of individually plausible figures that do not reconcile. 4–6 relations are assertable per document right now.
4. Deeper line items. Concepts the document presents that were never labelled — 74–96 per document.

Not counted in that projection, and both matter more than the headline

• Line items the parser misses entirely. A corpus that only labels what the parser already finds cannot measure recall, and recall is the number that moves when extraction regresses. The gap report is derived from what the pipeline emitted, so it is structurally blind to this. Authoring from the document's own statements — not from the gap report — is what closes it.
• Segment and disaggregation notes. The source-precedence rule makes the issuer's own segment note first-order, and it multiplies as segments × metrics × periods. 389 filing-sourced segment_revenue rows already exist to label against.

Schema work this implies

Levers 1, 2 and 4 need no schema change — author more rows. Lever 3 (derived/footing) needs one migration: extend the eval_corpus_labels_kind_chk CHECK with a derived_relation kind whose expected JSONB holds the relation expression and operand list, and extend eval_results_verdict_chk if a distinct does_not_foot verdict is wanted rather than reusing fail.

Maintained alongside src/admin/eval-corpus.service.ts. If behaviour and this page disagree, the code and the migration comments are correct — and this page is a bug.