Auditing a backtest for information it should not have had

The claim under audit

A multi-timeframe trend system had validated well across a ten-instrument portfolio, and eight of those instruments were cleared for deployment. On paper it was finished.

The audit existed to answer one question before any capital moved:

Does the backtest know anything at decision time that the live system could not have known?

That question is not about profitability. A backtest can be profitable and still be worthless, and the only way to tell the two apart is to reconstruct what information each decision was entitled to see.

The method

Line-by-line reconstruction of the information available at every decision point. Not a sanity check on the output — a reconstruction of the inputs.

The bug, once found, was quiet in exactly the way dangerous bugs are. Higher-timeframe trend windows were sliced by bar open time. That admitted the still-forming daily bar, carrying its final close, into the trend filter on every single evaluation. Because a daily bar’s eventual close correlates with the outcome of a breakout taken during that day, the filter was grading entries against the answer key.

Nothing about this produces an error. Nothing about it looks wrong in a results table. It simply makes the system appear to have judgement it did not have.

The step most audits skip

Finding a bug and fixing it changes the numbers. That is not the same as proving the bug caused the change.

So the old, biased slicing was re-emulated bit-for-bit on identical fresh data. It reproduced the original results to within a few hundredths of an R. That control run is what separates “the numbers changed” from “the cause is isolated” — without it, the correction could equally have been a shifted data window, a different sample, or a second unrelated bug.

An audit that cannot reproduce the flaw on demand has not diagnosed it.

What it cost

Most of the claimed edge was the bias. The win rate fell. Of the ten instruments, one survived the acceptance bar — and the eight that had been cleared for deployment were not cleared any more.

The invalidated benchmark was then retired in writing, so it could never be quoted as a target again. That matters more than it sounds: a number that has been disproven still circulates unless somebody records that it is dead.

What it became

The surviving instrument was rebuilt around what a follow-up exit study showed actually carried the edge, and revalidated on the corrected engine.

The bias class itself became a permanent audit item — including a check that the live scanner’s bar handling matches the corrected backtest’s semantics exactly. A backtest and a live feed can disagree about time in both directions, and a correction that fixes only the backtest just moves the discrepancy somewhere harder to see.

Why this is in a platform about provenance

This platform exists because of this audit, not alongside it.

Every constraint the platform enforces — figures traced to a hashed source file, metrics recomputed rather than transcribed, results graded by how they were produced, selection bias stated as a property of the data — is a generalisation of what this one audit had to do by hand. The reason the platform refuses to display an unlabelled number is that an unlabelled number is exactly what a biased backtest produces: something that looks measured and is not.