Published copy. Evidence links point at the repository on branch insights/eval-and-invariants.

Insights Goal A: split and land the evaluation system and the deterministic safety fixes

2026-09-07

Result: the landable branch insights/eval-and-invariants is pushed and its pull request is open. Nothing is merged or deployed. The regression gate did not pass its strict criteria. The control arm shows that the failed criteria are below the provider noise floor. No change was moved to the meaning layer by this gate. The meaning-layer changes are parked on insights/meaning-layer-parked.

Summary

• Classification: 90 eval-system paths, 162 invariant-fix paths, 73 meaning-layer paths in the post-goal-1 delta, plus 1841 evidence files (all eval-system). The goal-1 segment is already on origin/main.
• Gate run 2 (the first complete run): retained 26/36 original correct completions; precision 37/69 (53.6%) against the original 36/67 (53.7%); correct-to-wrong 5; correct-to-error 3; jobs at deadline 0/250.
• Gate run 3 (the one permitted full rerun, same code): retained 24/36; precision 33/71 (46.5%); correct-to-wrong 4; correct-to-error 5; jobs at deadline 0/250.
• Control arm: the original application, rerun with the same frozen method on the seven affected chains, retained 6/13 of its own correct completions (3 correct-to-wrong, 2 correct-to-error, 1 correct-to-refusal). Only 5/28 of its generated SQL statements matched its own baseline byte for byte.
• Fixture: pd_is_ct_employee now has 2 true rows. Control 39 changes the staff-exclusion answer from 64 to 62. The 39-control self-check passes on the corrected fixture and fails only control 39 on the frozen one.
• Checks on the final tree: full Python suite 28598 passed, 21 failed (all 21 pre-existing), 50 skipped. Ruff and strict mypy pass on src/. Node 244/244 on the goal-3 set. Chromium 56 passed (1.9m) and 15 passed (13.2s).
• Published report: https://ygee-1qdq-5awf-ce4.pages.dev. Pull request: https://github.com/stevanl/lore/pull/98.

Base branch note

Local main is 6ce7648a. origin/main is a922627d. origin/main already contains the goal-1 fix package e479e81 and one desk fix. The landable branch is built from origin/main, so the pull request shows only the post-goal-1 delta. The classification table still covers the full main..V18 range in two segments.

Step 1: classification

The full table is in classification.md. It lists every path, its group, a one-line reason, and the first commit that introduced it. Mixed files list which hunks went where.

Uncertain items and the rule applied to each are in the classification document. The main rule: a change that alters the SQL text that reaches the database, or the prompt that produces it, is meaning-layer even when it is deterministic.

Group-2 changes that depended on a group-3 helper

Step 2: the landable branch

Built from origin/main. The V18 tree was overlaid, then every meaning-layer module, hunk and test was removed. Six mixed source files were rebuilt hunk by hunk. Sixty test files are parked whole. Twenty-one mixed test files had their group-3 tests removed, or were restored to their base version when the V18 version only added group-3 expectations.

One overlay error was found and fixed before the gate: the overlay had reverted the four files of the later desk commit a922627d. They are restored to their origin/main bytes.

Checks on the final landable tree

Fixture control 39

Goal 3 found that pd_is_ct_employee had zero true rows, so the ratified staff-exclusion predicate removed nothing. The fix sets two synthetic Pipedrive people (ids 31013 and 31047) to true. No other value changes.

The corrected fixture is installed beside the frozen one, not over it. The frozen E5 seal binds the fixture generator and check scripts by absolute path and byte hash. The gate therefore ran against the frozen fixture with the sealed method bytes. The corrected fixture needs a new seal and a new baseline before it can anchor any paired claim.

Step 3: regression gate

Method: the frozen goal-3 E5 method. Same sealed corpus (250 attempts, 100 supported first turns, 50 conversations), same scorer and statistics, same fixture database, same clock anchor (2026-09-06 12:00 UTC, Europe/London calendar), provider glm-5.2 at temperature 0, seed 42, thinking off, two workers. The application snapshot is 7ba3a106 (source manifest a85de01c40aff655). The comparison arm is the retained original E5 run of e479e81. This is a regression gate, not an improvement claim.

Three candidate runs exist. Run 1 aborted after 25 attempts on provider timeouts caused by the R07 child transport, which was then parked (see below). Run 2 is the first complete run. Run 3 is the one permitted full rerun. Runs 2 and 3 use the same application bytes; no code changed between them.

Criteria

Outcomes

First-turn execution errors: 60/100 original, 54/100 run 2, 58/100 run 3. Correct first turns: 14/100 original, 16/100 run 2, 14/100 run 3. Generated SQL byte-identical to the original run: 153/250 in run 2, 152/250 in run 3.

Lost original completions, run 2

Lost original completions, run 3

Cause analysis of the run 2 losses

Each of the ten lost cases was traced through the provider trace, the stored turn rows and the application code on both snapshots.

• Seven of the ten cases sent a generation prompt whose SHA-256 was byte-identical to the original run's prompt. The provider returned a different completion. The other three are follow-up turns whose parent completion had already diverged on an identical prompt.
• Across the whole run, 34 of the 40 first-turn cases with identical prompts and no novel prompt produced a different final SQL statement. Provider glm-5.2 at temperature 0 and seed 42 is not deterministic.
• Deterministic code paths on the failing routes were compared on both snapshots with scoped tests: the SQL validator and its table allowlist (identical verdicts for the exact failing statements), the single-shot post-pass (identical output for the exact wrong follow-up statements), the CT-leg pre-dispatch gate, the regenerable-error marker list, the validation message text, the leg self-heal, and the worker error handlers. No difference was found.
• The two first-turn errors (deal_stage_timestamp_order:t1, activity_deadline_before_creation:t1) came from a gold leg that referenced an unqualified table. Both snapshots reject that statement at validation. In the original run the CT leg came back empty, the planner raised, and the single-shot path regenerated with a heal. In run 2 the CT leg came back with SQL, so the fan-out plan dispatched the bad gold leg and failed. The branch taken depends only on the provider's completion.

Control arm

To measure the noise floor, the original application e479e81 was rerun with the same frozen method on the seven chains that contain the ten lost cases (28 attempts, control arm). The candidate was rerun on the same chains (28 attempts).

The original application does not reproduce its own correct completions under the frozen method. The gate criteria of zero correct-to-wrong and zero correct-to-error transitions are below the provider noise floor. Per-case outcomes for all four samples:

Transitions, run 2

Transitions, run 3

The full 250-row tables are docs/evidence/insights-goal-a-2026-09-07/holdout/case-transitions-run2.json and case-transitions-run3.json.

Moved from group 2 to group 3 during the gate

One change moved: the R07 child-process provider HTTP transport (_JobDeadlineTransport, _job_http_client_kwargs and the child script in services/llm_executor.py, with its call sites in services/bi_sql_llm_generator.py). The first full gate run on snapshot d963f75e was stopped after 25 attempts: 20 of 29 provider calls timed out at exactly 25 s while the 7 that completed took 4 to 9 s, and a plain SDK probe answered in 1 to 4 s. Isolated reproduction: through the job-bound transport, 20 KB and 40 KB prompts complete in 2 to 3 s; 60 KB and 90 KB prompts time out at 25 s. Cause: Popen.communicate(input, timeout=0.025) times out before the child drains stdin, and the retry with input=None never re-registers stdin, so payloads above the pipe buffer never finish and the child blocks in json.load(sys.stdin). V18 showed the same defect as 10 timeouts in 906 calls; the base prompt path on this branch produces larger prompts, so it hit most calls. The transport is not meaning-layer, but it is defective, so it is parked with a fix note. The rest of N07 stays. The aborted run is retained at /tmp/lore-goal3-eval/holdout-goal-a-run-aborted-provider-timeouts and its receipts in the evidence tree. Run 2 (the first complete run) and run 3 (the one permitted full rerun) both used snapshot 7ba3a106 with fresh pristine states. The resume session moved nothing else: see the cause analysis and the control arm.

Nothing else moved. The resume session found no responsible change for the run 2 losses. Moving a deterministic fix to the meaning layer without a reproduced cause would not change the noise.

Gate verdict

The strict criteria fail on both complete runs. The control arm shows the same criteria fail for the original application against itself. The landable branch is therefore not shown to regress the original, and it is not shown to be equal to it either. The pull request is opened for a human decision with these numbers in its body. A gate that can pass needs a deterministic generator or a paired design with repeated samples per case.

Step 4: parked

Branch insights/meaning-layer-parked is created from the landable branch and applies group 3. It is not pushed. Its README, docs/plans/2026-09-07-meaning-layer-parked-README.md, lists what is there, why it is parked, the holdout numbers that parked it, the 12 lost original completions, and the 44 recovery cases from the goal-3 report.

Step 5: delivery

• Pull request: https://github.com/stevanl/lore/pull/98
• Landable branch head: 45ba8d5b
• Parked branch head: 481e8b92 (local only)
• Published report: https://ygee-1qdq-5awf-ce4.pages.dev (noindex, _headers with X-Robots-Tag: noindex)

Unverified

• Provider responses are not deterministic at temperature 0 and seed 42. Three samples of the candidate and two of the original disagree with each other on the same prompts. Any single-run number in this report has a wide noise band that this design does not measure.
• The gate ran against the frozen goal-3 fixture with the sealed fixture-method bytes. The corrected control-39 fixture has no sealed baseline yet; 5 of 250 holdout expected answers change under it (one family).
• The cause analysis rests on prompt hashes, stored turn rows and scoped tests of the deterministic stages on both snapshots. The exact provider completions of the lost cases were not retained, so the two first-turn errors were not replayed end to end.
• The native six-case browser path (Chromium to API to worker to PostgreSQL) was not rerun on the landable branch. The mocked Chromium checks and the live holdout are separate evidence.
• The E6 deterministic latency method was not rerun. No latency claim is made for the landable branch.
• The V18 server's effective runtime configuration is not retained locally as a file, so it was not byte-compared to this run. Both runs used the same sealed runtime script (hash 5eba0be0…) and the same seed fields; only the metadata database path differs by construction.
• The six parked Chromium clarification cases and 60 parked Python test files are not executed on the landable branch.
• Screen readers, physical touch, native Compose, and refreshed CSV after source changes remain unverified, as in goal 3.
• Pull request CI status was not observed after opening.

Evidence

• gate-run2.json — Gate criteria and numbers, run 2, with the control arm.
• gate-run3.json — Gate criteria and numbers, run 3.
• case-transitions-run2.json — All 250 case transitions, run 2.
• case-transitions-run3.json — All 250 case transitions, run 3.
• admission.json — Run 2 server admission: snapshot manifest, method hashes, definition probe.
• admission.json — Run 3 server admission.
• admission.json — Control arm admission (original snapshot).
• admission.json — Candidate scoped rerun admission.
• aborted-run-1 — Run 1 receipts (aborted on the R07 transport defect).
• fixture-data.diff — The two changed fixture rows.
• fixture-self-check.json — 39-control self-check on the corrected fixture.
• fixture-self-check.json — 39-control self-check on the frozen fixture (control 39 fails).
• holdout-expected-answer-drift.json — Expected answers that change under the corrected fixture.
• recovery-cases-44.json — The 44 dependent follow-ups after a failed or refused parent.
• checks — Full-suite, curated-binding, Node and Chromium receipts.
• classification.md — Classification table.
• classification.json — Classification data.