The selected result carries a search history
A strong-looking result is less surprising when many alternatives were examined. Research trials include more than named models: changing a lookback, a universe filter, a score weight, a rebalance date or an exit can create another opportunity to select noise. A publication that reports only the winner conceals information needed to judge how exceptional that winner really is.
Harvey, Liu and Zhu examine multiple testing in the cross-section of expected returns, while Bailey and coauthors study the probability of backtest overfitting. These primary research papers address related selection problems using different frameworks. They do not supply a universal pass mark for every equity study, and neither replaces accurate data, a meaningful economic hypothesis or credible implementation assumptions.
A simple probability calculation shows the issue
Under an intentionally simplified model, suppose each of 20 independent tests has a 5% false-positive probability and every tested hypothesis is actually null. The probability of at least one false positive is one minus 0.95 to the twentieth power, approximately 64.15%. With 100 such tests it is about 99.41%. A conventional single-test threshold does not control the whole search.
Real strategy variants are usually dependent, so these numbers are not estimates of a particular research project's error rate. Nor does a significant result automatically have a 5% probability of being false. The example concerns the chance of at least one rejection under specified null assumptions. Preserve that distinction when explaining statistics to readers or investment committees.
Record the decision tree, not only saved models
Maintain a trial register containing hypothesis, data version, sample dates, feature set, ranking objective, construction, exit rule and cost assumptions. Record abandoned variants and negative results, including informal trials that changed the eventual design. Four windows, three selection thresholds, two weighting rules and two schedules already create 48 combinations before considering further adjustments.
Choose the selection criterion before comparing candidates. If the experiment concerns entry information, do not quietly rank entries by whichever exit produces the best profit factor. Freeze the entry identity and the predeclared exit treatment, then use economic outcomes for the declared validation question. Reopening an exit tournament after viewing the evaluation period expands the search and consumes more of its independence.
Use correction and fresh evidence together
A multiple-testing adjustment should match the hypotheses and dependence structure actually examined. A complete trial history makes such analysis more credible; an unknown history should be disclosed rather than assigned an invented effective trial count. Examine parameter neighborhoods and predeclared subperiods, but remember that selecting the most flattering robustness display is itself another selection decision.
Keep a later evaluation period outside the selection process and state when it was first viewed. Corrections can quantify aspects of selection uncertainty; they cannot rescue revised data masquerading as historical information or impossible fills. Software unit tests check calculations, not economic independence. No candidates were fitted for this guide, and its probability table is mathematical illustration rather than a measured overfitting probability for an investment model.
Inspect the example
| Independent null tests assumed | Per-test false-positive probability | Probability of at least one false positive |
|---|---|---|
| 1 | 5% | 5.00% |
| 20 | 5% | 64.15% |
| 100 | 5% | 99.41% |
Sources & reproduction
Primary references support the definitions and method. Numerical examples are illustrative unless a source record is explicitly identified.