A strategy that buys the London open and holds four hours will look roughly the same in any tester you point at it. A strategy with a 6 pip stop will look completely different depending on whether the engine models the spread widening at 22:00 server time, and whether the intrabar path is reconstructed from ticks or guessed from the open, high, low and close.
That gap is where most of the disappointment in this field comes from. Choosing a tool is really choosing how much of the real cost structure you are willing to simulate.
MetaTrader: the default, with caveats
The MT4 tester was built for single-symbol expert advisors and reports a modelling quality percentage that is honest about what it did. Anything below full tick modelling is producing fills the market may never have offered. Most of the historical data supplied with a broker terminal is minute bars, and the tester interpolates ticks inside them unless you import proper tick history.
The MT5 tester is a different piece of software. It handles multiple symbols in one pass, supports real tick data where the broker provides it, models the actual contract specification including swap and commission, and includes a genetic optimiser with distributed agents. For anyone testing an expert advisor that will run on MetaTrader in production, this is the correct tool, and the MT5 strategy tester guide covers the settings that matter.
The caveat applies to both. The test uses the symbol specification of the broker whose terminal you are running. Change broker and the spread profile, swap rates and stop level restrictions change with it, which is enough to move a marginal strategy from profitable to not.
TradingView: fast to write, easy to fool
Pine strategies are the quickest route from an idea to an equity curve. You write the entry and exit conditions, add commission and slippage in the properties panel, and get a report in seconds. For swing strategies on daily and four hour bars this is a reasonable answer, and the Pine basics are learnable in an afternoon.
The problem is intrabar resolution. By default the engine evaluates on bar close and assumes an order path inside the bar, so a strategy whose stop and target could both be hit within one candle gets an assumption rather than a measurement. Higher plans offer finer intrabar granularity, and using it changes results on short-term systems noticeably. Anything that repaints, including signals computed with future-referencing offsets, will produce a backtest that cannot be traded.
| Tool | Best for | Main weakness |
|---|---|---|
| MT4 tester | Legacy EAs on one symbol | Interpolated ticks unless imported |
| MT5 tester | EAs, multi-symbol, optimisation | Tied to one broker's specification |
| Pine strategies | Fast idea screening on higher timeframes | Intrabar assumptions, repainting risk |
| Manual bar replay | Discretionary methods | Slow, and easy to cheat yourself |
| Python frameworks | Portfolio logic, custom cost models | You build and validate everything |
Manual replay for discretionary methods
If a method depends on reading structure, sessions and context, no code will capture it and there is no point pretending otherwise. Bar replay in a charting package, or a dedicated replay simulator, lets you step forward candle by candle and take the trade before you can see the outcome.
The discipline requirement is absolute. Write the entry, stop, target and size into a journal before advancing a single bar. Do not skip periods that look uninteresting, because avoiding chop is exactly the behaviour that makes a live account look nothing like the test. Two hundred trades collected this way tells you more about a discretionary edge than any optimiser output.
A backtest measures the interaction of a rule set with a cost model. If the cost model is wrong, the result is not a weaker version of the truth, it is a different question answered confidently. Put real spread, real commission and real swap in before you look at the curve.
Python when the question gets specific
Frameworks like the widely used open-source event-driven and vectorised libraries earn their place once the question stops fitting a platform. Portfolio-level rules across twenty symbols, correlation-aware position sizing, a custom slippage function that scales with measured volatility, monte carlo resampling of the trade sequence: these are straightforward in Python and awkward everywhere else.
The trade is that you now own the correctness of the engine. Look-ahead bias creeps in through a single misaligned index. Costs have to be modelled by hand. Data has to be sourced, cleaned of bad ticks and aligned to the right session boundaries. It is worth it for research, and it is more work than most traders expect before they start.
The failures that no tool prevents
Optimisation is the big one. Run enough parameter combinations and something will look excellent on any dataset, including random data. The defence is to hold out a period the optimiser never sees, then walk the parameters forward across successive windows, which is the subject of overfitting in backtests. If the out-of-sample result collapses, the in-sample result was a description of noise.
Survivorship and specification drift matter more than people think in leveraged products. Contract sizes, margin requirements and available instruments change over the years. A test over eight years of gold history at today's contract specification is describing an instrument that did not exist for most of that period.
The last one is sequencing. A backtest tells you what a rule would have produced with perfect execution and zero emotion. What happens next belongs to forward testing on a demo or small live account, where latency, requotes and your own reaction to a five-loss run enter the data for the first time. Trading leveraged products carries a high risk of loss, and a good historical result does not reduce that risk in the next period.
"Show me the cost assumptions before you show me the equity curve. If the spread is fixed at one pip and slippage is zero, I already know what the live version looks like."
— Alex Onta, Executive Director, SINGUARD
Key Takeaways
- Tick quality and the cost model drive accuracy far more than which package you choose.
- MT5 suits expert advisors that will run on MetaTrader, Pine suits fast screening on higher timeframes.
- Manual replay is the only honest option for discretionary methods, and only if trades are recorded before stepping forward.
- Hold out data the optimiser never sees, then walk parameters forward before trusting any result.
Frequently Asked Questions
Which backtesting tool is most accurate?
Accuracy comes from the data and the cost model rather than the software. A tick-level test with real spread and commission in MetaTrader or a custom Python engine fed with quality tick data will beat a bar-level test anywhere, and a bar-level test on daily candles is only useful for slow strategies.
Why does my backtest look better than live trading?
The usual causes are fixed spread assumptions, no slippage, fills at the exact level, missing swap costs, and parameters tuned on the same data used to measure the result. Each is worth a small amount on its own and together they can invert the outcome.
Is manual bar replay a real backtest?
It is a valid way to test a discretionary method, provided you record every decision in advance and never scroll forward before writing the entry, stop and target down. It is slow and it is the only option for methods that cannot be expressed in code.