Who this is for — Anyone turning a trading idea into a controlled simulation who needs to distinguish what the model could actually have known and traded from what becomes visible only in hindsight.
A backtest applies a complete specification of signals, portfolio rules, orders, costs, and constraints to historical data. It produces a counterfactual result: what would have happened under that model if the rules had been executed during the observed period. It is therefore a conditional simulation: its outcome is conditional on the specified data, rules, execution model, costs, and constraints. It is not a live track record, it does not establish a causal relationship, and it does not guarantee that the future distribution will resemble the past.
The critical distinction is between the strategy and the simulator. The strategy says when to take risk; the simulator turns that decision into positions and returns. A sound signal can appear exceptional because of a timestamp error, untradeable prices, or omitted costs. Conversely, an excessively punitive execution model can conceal the behavior under study. Assumptions must therefore be stated and open to verification.
What a backtest must specify
| Layer | Verifiable question | Error it helps prevent |
|---|---|---|
| Data | Which value was available at that exact time? | Look-ahead, revisions, survivorship bias |
| Signal | Which variables, lags, and parameters determine the decision? | Rules reconstructed after seeing the result |
| Portfolio | How do signals become weights, sizes, and limits? | Implicit leverage or concentration |
| Execution | When and at what price can the order be filled? | Impossible same-bar fills |
| Economics | Which costs, funding, borrow, and corporate actions apply? | Treating gross performance as achievable |
| Evaluation | Which metrics, benchmarks, and tests were chosen beforehand? | Selecting the most favorable interpretation after the fact |
Data must be point in time whenever their history can change. This includes index membership, delisted securities, fundamentals with publication dates, revised macroeconomic series, and corporate classifications. Including dead tickers is not enough when delisting returns, historical membership, or the correct treatment of corporate actions are missing.
Causality: know, decide, execute
Every event should have at least an availability time and an action time. If a bar close enters the signal, a fill at that same close requires a credible specification of the auction, computation time, and order submission; otherwise, the first usable price belongs to a later event. The same principle applies to releases, economic indicators, and fundamental data: the reference date is not necessarily the date on which the information becomes public.
Futures, options, bonds, short positions, and leveraged instruments also require coherent rules for expiries, rolls, multipliers, margin, exercise, assignment, borrow, collateral, and the return on cash. No single fill model is valid for every asset class.
Minimum reproducible protocol
- Define the question. State the economic hypothesis, universe, frequency, period, benchmark, and the criterion used to judge the result.
- Freeze the specification. Version signals, parameters, sizing, constraints, data, and calendar. Keep a ledger of all trials, including failed ones.
- Build the event clock. Order information availability, decision, order submission, possible cancellation, and fill.
- Simulate the portfolio. Apply capital, leverage, liquidity, turnover, and concentration limits without using future information.
- Estimate execution. State spread, fees, slippage, impact, partial fills, funding, and relevant opportunity cost.
- Measure without choosing after the fact. Report net return, risk, drawdown, turnover, exposure, capacity, and benchmark comparison, together with statistical uncertainty.
- Validate separately. Reserve temporal data not used in development and specify how dependence, overlapping labels, refits, and final tests will be handled.
- Archive the artifacts. Preserve code, configuration, dependency versions, seeds, data snapshots or hashes, and primary outputs.
Costs and reference prices
Fees and spread are only part of cost. A simulation may also need to account for market impact, available quantity, delay, non-execution, funding, securities lending, currency conversion, and taxation relevant to the stated scope. Implementation shortfall compares execution with the intention at the decision time and can include the unexecuted portion; it is not equivalent to adding a fixed commission at the end of the calculation.
Where data allow, the cost model should depend on side, size, liquidity, volatility, venue, and order type. A more severe cost scenario is informative, but it cannot repair contaminated data or impossible fills.
Illustrative example
Illustrative example, not an operating result — A daily rule computes its signal after the close. The protocol uses point-in-time constituents, submits the order in the following session, caps participation in volume, and charges spread, fees, and an impact that increases with size. Net metrics are compared with a total-return benchmark in the same currency. If the researcher tries twelve lookback variants, all twelve enter the trial ledger: presenting the sole winner as the original hypothesis would be incorrect.
The example does not establish suitable lags or costs for a particular market. It shows which components must be open to challenge and recalculation.
How to read the result
An upward curve is not validation. Ask whether returns are concentrated in a few events, whether they change under small and reasonable perturbations, whether they survive plausible costs, whether the benchmark was selected ex ante, and whether estimated precision is consistent with the amount and dependence of the observations. The number of trades is not automatically the number of independent observations.
A backtest is more useful as an instrument of falsification and diagnosis than as a machine for producing a point forecast. A negative result may expose an incoherent hypothesis; a positive result makes the strategy a candidate for robustness checks, out-of-sample validation, and forward observation. It does not turn the strategy into a certainty.
Limitations
- The observed past may not contain future crises, regimes, or market microstructures.
- Apparent quality depends on the number of models and parameters explored.
- Slippage and impact are counterfactual and become more uncertain with size.
- A rerun with the same code and data tests reproducibility; it is not an independent replication.
- Monte Carlo and stress tests describe constructed scenarios. They do not remove look-ahead, survivorship bias, data snooping, or specification errors.
Sources
- Halbert White, A Reality Check for Data Snooping, Econometrica, 2000 — inference when the best model emerges from many explored specifications.
- Robert D. Arnott, Campbell R. Harvey, and Harry Markowitz, A Backtesting Protocol in the Era of Machine Learning — research protocol, selection, and the scarcity of financial data.
- Marcos López de Prado, What to Look for in a Backtest — number of trials, test length, and characteristics that should be disclosed.
- André F. Perold, The Implementation Shortfall: Paper Versus Reality, The Journal of Portfolio Management, 1988 — comparison between a paper portfolio and effective execution.
- European Commission, Commission Delegated Regulation (EU) 2017/589, RTS 6 — methodology, documentation, and testing environments for algorithmic systems within its regulatory scope.