What the Test Set Was Allowed to Change
A held-out dataset becomes development evidence when its result is used to revise the procedure.
The evaluation ledger has three entries. First, evaluate the frozen procedure once on held-out simulated engines. That result can evaluate that procedure. Second, change a threshold after reading it. Third, evaluate the revision on the same engines. No row entered the optimizer, but the result helped produce the revision. For the revised procedure, the set is now development evidence, not clean test evidence.
Training, validation, and test are roles in an adaptive process. Training estimates the state of a candidate. Validation or inner resampling chooses among candidates and may set thresholds. Test evidence evaluates the chosen procedure after the candidate, threshold, metric, slices, and acceptance rule are frozen. A filename cannot enforce those roles. In Python terms, the variable name does not determine the dataset’s evidentiary role. The selection workflow does.
This is adaptive reuse: a later choice depends on an earlier evaluation. When many alternatives are compared through the same holdout, the winner can benefit from genuine signal and from favorable peculiarities of that holdout. The more decisions the holdout is allowed to influence, the more opportunity its accidents have to enter the selected procedure. The human-and-code loop can learn the test set even when the fitting code never trains on its rows. A better result on test_final_v2 can therefore support a weaker claim than a lower result on evidence that remained untouched.
The failure mode is ordinary, not scandalous. An evaluation reveals a weakness, and the analyst does the reasonable thing: revise the procedure. The problem begins when the revised score is presented as though the evidence never participated in that revision. A holdout can be ceremonially present and evidentially absent.
Make the boundary concrete. At cycle 100 of a simulated engine trajectory, six available sensor readings are used to predict whether the trajectory will reach its terminal cycle within the next 75 cycles. Completed endpoints may construct retrospective labels, but they may not enter the scoring features. Engines, not rows, are assigned to evaluation partitions. If a held-out result changes the feature list, threshold, subgroup policy, or candidate, those engines have become development evidence for the revision. Fresh untouched evaluation evidence is now required for a new confirmatory claim. Without it, the revised result must be reported as exploratory or as a narrower descriptive result.
At most, fresh held-out evidence can support predictive performance on new simulated engines from the represented simulation regime. It does not show readiness for operating aircraft, the causal effect of a maintenance intervention, or field validation or certification. The ledger should say what happened: reused test evidence became development evidence for the revision. Either obtain fresh evidence for confirmation or label the reused result as exploratory and narrow the claim.
Read the complete argument on lospino.so:
The Model Is the Evaluation Protocol
Original essay published on lospino.so on July 17, 2026. This Substack dispatch is an adapted pointer to the canonical version, not a mirrored copy.