Someone runs an optimiser overnight on two years of EURUSD. In the morning the best result has a moving average length of 47, an RSI threshold of 68, a stop of 23 pips, a target of 61, entries allowed only between 08:00 and 11:00, and a filter that skips Wednesdays. The equity curve climbs at a beautiful angle. The strategy is then switched on live and immediately behaves like a coin toss with costs attached.
Nothing broke. The system did exactly what it was built to do, which was to describe two specific years of price data in as much detail as its settings allowed. Skipping Wednesdays was not a market insight. It was the optimiser removing a handful of losing trades that happened to land on that day.
Where the fit comes from
Every adjustable setting is a chance to bend the model toward the sample. With one setting you are testing an idea. With eight you are drawing a shape around the data you already have, and the shape will fit, because there is always some combination that does.
Two numbers decide whether a result means anything: how many things you tuned, and how many independent trades the sample contains. Six settings fitted against ninety trades is not evidence. The same six settings against several thousand trades across multiple instruments and regimes starts to be. If the trade count is small, drop settings until it is a strategy simple enough that the sample can actually say something about it.
Repeated testing does the same damage more quietly. Testing forty variations and reporting the best one is a single test with a hidden selection step, and the winner's result is inflated by the selection. If you looked at forty, the honest number to report is the average of the forty, not the top of the list.
Data problems that look like edge
Half of the results that fall apart live were never real in the test either, because the test data or the test assumptions were wrong.
- Fixed spread assumptions. Testing at a constant one pip and trading through news, the session open or the rollover window produces a very different cost line. Spread widening is a real and recurring expense.
- Bar data used for intrabar logic. If both your stop and your target sit inside the same candle, a bar-based tester has to guess which came first, and testers usually guess in the direction that flatters the result.
- No slippage and no commission. A strategy averaging a few pips per trade lives or dies on these. Add realistic costs before drawing conclusions, not after.
- Missing or synthetic ticks. Free history is frequently patched. A gap filled by interpolation can invent price paths that never happened.
- Symbol history from a different feed. Testing on one broker's data and trading at another means the levels themselves differ slightly, which matters for anything triggering on precise prices.
Getting these right is largely a matter of choosing an environment that models them, which is what our comparison of backtesting tools covers. A tester that supports real tick data and variable spreads costs more effort to set up and returns a far less exciting number, which is the point.
The tests that expose it
Out-of-sample testing is the first and weakest defence. Hold back the final third of your data, tune on the rest, then run the frozen settings on the part you never looked at. It is weak because after the first failed attempt most people go back, adjust, and run the reserved data again. At that point the reserved data has been used for tuning and is no longer out of sample.
Walk-forward analysis is stronger. Optimise on a window, trade the next window with those settings, roll forward, repeat, and judge only the traded windows. It answers the question that matters: would this process, applied as you would actually apply it, have produced money in periods it had not seen.
Then look at the parameter surface instead of the parameter. Plot results across the range of each setting. A genuine edge shows a broad plateau where neighbouring values all work reasonably. A fit shows a single spike, where 47 is excellent and 45 and 49 are losses. That spike is the clearest single sign of overfitting available to a retail trader, and it takes minutes to check.
If changing one setting by a small amount destroys the result, you have not found a rule the market follows. You have found a coincidence in a particular stretch of history. Prefer the wide plateau even when its headline number is lower.
What passing actually looks like
A result worth taking further usually has a modest number of settings, a trade count in the hundreds rather than the dozens, similar behaviour across at least two instruments or two years with different character, and stability when settings move. It also has a drawdown you can explain in terms of the strategy's logic rather than one that appears from nowhere.
What it does not need is a spectacular return. Simple systems with fewer settings tend to degrade far less between test and live than optimised ones, and the gap between a backtest and reality is the only number that matters once real money is involved.
After the tests, the last step is time. Freeze the ruleset and run it forward on data that did not exist when you chose the settings, either on a demo account or at minimal size. That is the one check hindsight cannot contaminate, and it is also where the limits described in paper trading become relevant, because a forward test proves the logic while telling you nothing about how you will behave when the money is yours. Keep the results in a proper journal so the comparison is written down rather than remembered.
The same discipline applies to anything you buy rather than build. A vendor's equity curve is an optimised backtest until proven otherwise, and the questions in our guide to expert advisors are the ones to ask before paying for one.
"If moving one setting by two points turns a great result into a losing one, you did not find an edge. You found a coincidence, and coincidences do not repeat on request."
— Alex Onta, Executive Director, SINGUARD
Key Takeaways
- Judge a backtest by the ratio of tuned settings to independent trades, not by the shape of the equity curve.
- Plot the parameter surface. A broad plateau suggests a real effect, a single spike suggests a coincidence.
- Model costs honestly: variable spreads, commission, slippage and real tick data change the answer for short-target systems.
- Reserved data stops being out of sample the moment you tune against it. Forward testing on genuinely new data is the only clean check.
Frequently Asked Questions
What is overfitting in a trading backtest?
Overfitting is when a strategy has been tuned so closely to one historical dataset that it describes the noise in that data rather than any repeatable market behaviour. The result looks excellent on the sample it was built on and behaves like a random system on data it has never seen, because the settings were chosen to fit accidents of the past.
How many parameters is too many?
There is no fixed limit, but the ratio between the number of settings you tuned and the number of trades in the sample matters more than either figure alone. Six adjustable settings fitted against ninety trades leaves very little room for the result to be anything except a fit. Fewer settings tested against more trades is the direction that produces conclusions worth trusting.
Does forward testing prove a strategy works?
It cannot prove it, but a failed forward test is decisive in the other direction. Running the frozen ruleset on data that did not exist when the settings were chosen is the only test that cannot be contaminated by hindsight. A strategy that survives several months of forward testing is worth a small live allocation. Trading remains high risk regardless of how a system tested.