RQ2 ActPlane False-Positive Attribution

Status: completed trace audit of the primary Qwen artifact, 2026-09-09. This is an evidence note, not a change to the submitted rebuttal.

Method

The audit examines all 18 rows labeled FP for ActPlane in the frozen origin/artifact-ready manifest. Each attribution uses the original directive, frozen rule.yaml, compliant trace, runner result, feedback target, paired opaque result, and judge rationale. Primary categories identify the first stage that made the compliant trace receive an incorrect intervention. Secondary categories retain contributing evaluation effects. Counts are descriptive causal audit results, not new model judgments.

The historical matcher behavior is evaluated at the experiment commit lineage, not inferred from current code. In the 2026-06-07 compiler, repo-relative **/*.js lowered to CONTAINS .js, **/dist/** required the substring /dist/, a repo-relative pattern containing * lowered to its prefix before *, and a repo-relative basename such as .env.test lowered to CONTAINS. Some of these lowerings have since changed, so the table does not assign historical output to the current compiler.

Complete 18-trace attribution

Attribution totals and interpretation

Primary causes are 9 translation omissions or deliberate over-approximations (#6, #7, #9, and #13 to #18), 4 historical path-lowering over-approximations (#2, #3, #8, #10), 4 harness-tool interactions (#1, #4, #5, #12), and 1 trajectory protocol/judge error (#11). These categories cover all 18 traces exactly once.

Seventeen judge FP labels are supported under the paper's end-to-end definition, although three rationales conflate a harness or recovery subprocess with the user-requested action. Case #11 is not supported as a policy false positive because ActPlane did not intervene on the compliant setup. It intervened only after the recovery agent issued a newly noncompliant bare-Python command.

The original “17 of 18 are caused by broad translation” sentence is therefore not supported by this audit. At most 13 involve translation or the historical lowering of translated path patterns, while four arise at the harness boundary and one is a judge/protocol error. Even the 13 should not be collapsed because policy semantic coverage and compiler matching precision are different system stages.

Independent local-model review

An independent Qwen3.8-27B review resumed the interrupted OpenCode session and reached a final stop after reading the same frozen evidence. It independently returned 9 translation, 4 lowering, and 5 harness/environment cases. That agrees with 17 row-level primary attributions and assigns #11 to the harness/environment category rather than the protocol category used here. It also considers #11's FP label supported because a recovery kill occurred. This note retains the disagreement instead of forcing consensus: the raw evidence clearly establishes that setup had no ActPlane trigger and recovery used forbidden bare Python, while the published judge definition is ambiguous about whether a newly violating recovery action makes an originally compliant trace FP. The paper should specify that semantic choice before reporting a definitive label for this row.

The independent review also confirmed that 14 paired opaque notifications match without becoming FP, the shared opaque FP is the unavoidable kill in #8, and the remaining three pair differences are separate-execution variance. Its complete machine transcript is /tmp/opencode-rq2-audit-resumed.jsonl; it is supporting review evidence, not a substitute for the frozen runner and judge records.

Why ActPlane has 18 FP and opaque has 1

Fifteen corresponding opaque executions also record a setup trigger. In fourteen of those, the effect is a hidden notification, so the opaque agent is neither blocked nor discouraged and the end-to-end judge returns TN. The one opaque FP is case #8, where the effect is kill and cannot be hidden. Two harness-Python cases do not trigger in the separate opaque executions, and case #11 has no setup trigger in either mode. Therefore the 18-to-1 difference is mostly the expected consequence of making notifications visible under a metric that defines discouragement as FP. It is not evidence that semantic feedback changes the same kernel rule into one that matches 17 additional compliant traces.

Evidence locations and reproduction

The machine-readable stage audit is produced by:

node docs/empirical-study/audit_rq2_verdicts.js ARTIFACT_ROOT audit.json

For each table row, audit.json gives result_file. Its sibling trajectory_judges_llama_cpp_guardrail_response/<run>.judge.json is the judge record. The frozen rule and trace are at ARTIFACT_ROOT/docs/corpus-test/<repo>/<statement>/rule.yaml and the named trace JSONL. Historical lowering is at cc3a9b11:collector/src/dsl/lower.rs, which is the repository state associated with the 2026-06-07 result lineage. The current implementation must be evaluated separately before claiming that any historical lowering defect remains.