Who this is for — Anyone developing or evaluating systematic strategies who needs to distinguish a measurement obtained on genuinely unseen data from a continuation of the optimization process.
An out-of-sample (OOS) test evaluates an already defined specification on data that did not contribute to the choice of its rules, variables, parameters, or selection criteria. Its purpose is to measure the ability to generalize beyond the development sample. It does not prove that the behavior will continue in the future: it reduces one particular form of optimism, provided that the separation is genuine.
OOS purity concerns the flow of decisions, not merely a column of dates. If the test result causes the model to be changed, that segment has become development information. The modified version needs another test that has not been inspected. It is incorrect to adjust the strategy and continue calling the same data out of sample.
In-sample, validation, test, and forward are not synonyms
| Block | Permitted use | What contaminates it |
|---|---|---|
| Training / in-sample | Estimate and fit parameters | It is not intended as an impartial evaluation |
| Validation | Select among specifications and hyperparameters | Repeated consultation is allowed but must be counted |
| Final OOS test | Conclusive evaluation of a frozen specification | Any change guided by its result |
| Walk-forward | Sequence of fits and tests ordered through time | Rules redesigned after seeing the full sequence |
| Forward or paper test | Observation later in time, often without real money | Differences among simulated fills, operations, and live capital |
An OOS sample can be a final holdout, a window in a rolling procedure, or part of temporal cross-validation. The name alone does not specify the design. Window sizes, ordering, label horizon, refit frequency, and the total number of decisions made from the results must be documented.
There is no universal split proportion
There is no general rule such as 70% for development and 30% for testing. The choice depends on the quantity and quality of data, frequency, forecast horizon, process stability, number of parameters, and the need to observe different regimes. A very long test reduces the development data; one that is too short may yield imprecise estimates or represent a single regime.
Order matters in time series. A standard random split can place observations in training that are closely connected to those in the test. It should not, however, be declared universally invalid: its appropriateness depends on the problem and on assumptions about dependence. In finance, autocorrelation, window-built features, and overlapping labels often call for blocked or rolling splits or other specific separation techniques.
Purging and embargo
When a label uses a future interval—for example, the return between opening and closing a position—a training observation can share part of that same information interval with a test observation. Purging removes from training the observations whose label intervals overlap the test. An embargo adds a temporal buffer to reduce residual dependence around the boundary.
The buffer must be justified by the horizon, data availability, and dependence structure; a fixed percentage is not a general principle. Purging and embargo do not correct survivorship bias, revised data, unrealistic costs, multiple testing, or regime change.
Validation protocol
- Define the information unit. Specify feature timestamps, label start and end, frequency, and possible overlaps.
- Separate roles. Assign data to training, validation, and test before seeing final metrics. Do not call a block used for tuning a test.
- Choose the design. Justify the holdout, rolling origin, expanding or rolling window, blocked cross-validation, and any purge or embargo.
- Freeze the pipeline. Version transformations, features, universe, hyperparameters, costs, benchmark, and decision rule.
- Record every trial. Count models, seeds, periods, filters, and metrics inspected, not only the published variant.
- Run the final test. Calculate net metrics, uncertainty, and benchmark comparison without reopening selection.
- Classify the outcome. “Inconclusive,” “inconsistent with the hypothesis,” and “compatible with further testing” are more accurate than pass/fail based on a universal threshold.
- Document every reuse. If the result informs a change, formally move that period into the development record.
Walk-forward and the final test
In walk-forward evaluation, the model is estimated using only data before each test window. The procedure emulates successive refits and shows how the result varies over time. It must state whether the training window expands or rolls, the step size, the tested horizon, and the recalibration schedule.
Concatenating the windows does not automatically create a pristine track record. If the model family, features, windows, or stopping rules are changed after inspecting the full sequence, that sequence has participated in development. Where possible, an internal selection procedure can be combined with an external final holdout that remains unseen.
Illustrative example
Illustrative example — A monthly model is refitted with a moving window and evaluated on the following month. Position labels can extend over several days, so before each test the training labels that intersect its interval are removed. The researcher compares four model families in internal validation and records all four. A later temporal block remains sealed for the final decision. The durations are features of the example, not recommended proportions.
The result should be reported with dispersion across windows, costs, and exposures, not only as an aggregate metric. A favorable mean can conceal that nearly all value came from one period.
Limitations
- A single holdout can be heavily influenced by the regime it contains.
- Multiple data reserves do not create new independent information.
- Non-stationarity can make both training and test unrepresentative.
- The validation design is itself a research choice and must enter the trial ledger.
- OOS, purging, and walk-forward do not replace point-in-time data, causal simulation, plausible costs, and control of data snooping.
Sources
- Leonard J. Tashman, Out-of-sample tests of forecasting accuracy: an analysis and review, International Journal of Forecasting, 2000 — rolling origin, recalibration, and multiple test periods.
- Christoph Bergmeir and José M. Benítez, On the use of cross-validation for time series predictor evaluation, Information Sciences, 2012 — cross-validation designs for time-series predictors.
- Christoph Bergmeir, Rob J. Hyndman, and Bonsoo Koo, A note on the validity of cross-validation for evaluating autoregressive time series prediction, 2018 — conditions under which cross-validation can be valid for autoregressive series.
- David H. Bailey et al., The Probability of Backtest Overfitting — selection among configurations and the limitations of a simple holdout in investment backtests.
- Marcos López de Prado, Advances in Financial Machine Learning — official Wiley table of contents, Wiley, 2018 — monograph reference for purged K-fold and embargo; it is not cited here as a peer-reviewed article.