I was torn on whether or not to write this, but the need for transparency kicked in at the last moment so here goes.
We found it. It was the perfect strategy. It was correct 95% of the time. It had an annual drawdown of less than 1%. It made over $200K on a single contract in one year.
We were ecstatic and felt like kids the night before Christmas. We’d set out to find the perfect automated strategy and we did!
The next day we put the strategy to the test on a simulated market.
It performed horribly.
We use Ninjatrader 7 to backtest strategies so we reached out to NinjaTrader Support to see if this was a known issue and the answer was…yes.
I felt like I was being impaled by a unicorn. I mean why have a backtester if it doesn’t really work?
I contacted Ninjatrader support for additional help. This, in a nutshell, is what they said:
1) Historical data used for backtesting does not include all bar data. It knows the range of prices for the bar, but not the timing of the price.
2) Historical data used for market replay does include this granularity so the results from market replay are more in line with actual results.
3) Some strategies will work on the backtester, especially those with an average trade time of 1 minute or more. However, those using a limit order to place the trade (20 ticks) and a tight take profit (20 ticks), will always over-perform in the backtest.
You can read their full explanation here.
So, in case you’re still curious about the strategy. Here you go:
Enter Long - MACD crosses below MACD average with a 20 tick limit order.
Enter Short - MACD crosses above MACD average with a 20 tick limit order.
Take profit - 20 ticks
Stop loss - 100 ticks
Notice this is counterintuitive — most strategies enter a long on the cross above, not below and vice versa. The goal was to take advantage of the pull back or volatility that usually occurs when MACD crosses MACD average (up or down). Again, this strategy does not work in real time, but since I’ve talked about it, I thought you might be interested.
I like to think there’s always something redeemable in an exercise. What did this exercise teach us?
Limit orders can be powerful.
Backtesting is only reliable if the take profit is more than 3 to 4 times the limit in ticks (at least on NT 7).
The further away you can afford to place your stop loss, the more likely the take profit will be hit. This may seem fairly obvious, but it’s worth saying.
Moving forward, we’re using the lessons learned here to continue on our search for the holy grail of trade strategy. Evidently NT 8 has a more robust backtesting apparatus so we’re considering an upgrade to NT 8.
Finally, and perhaps most importantly, going forward we will not publish any strategies unless we’ve 1) used them live and 2) backtested using market replay, not the backtester.
As a final note, none of the nine strategies we’ve published so far use limit orders or take profit orders (unless optimized) so this does not impact those results.