Over the last 12 months our top strategies made over $2.8M based on NT8 backtest results. Average profit factor increased from 1.40 to 2.02 since the May update.
It's a fair question. And ultimately, I made the final decision on this one.
The short answer is: If I trusted the data more, I would, but I don't.
The long answer is: Knowing that a strategy performed well for the last 5 years as opposed to the last year would certainly make me 'feel' better about trading it live, but just because you run a 5 year backtest doesn't mean you're going to get the answer to that question. One of our primary concerns is backtest risk or the accuracy of the standard backtest. Standard backtests use historical data, which is inherently flawed and I believe long dated backtests can exacerbate the issue. This is because the data tends to deteriorate the further back you go. Even if the strategy performs better in the past than it does in the current year, I wouldn't trust it. Unfortunately, we've been impaled by a few of our own unicorns already, which is to say, we know how inaccurate a backtest can be (If you follow the newsletter, you know what I'm talking about). To avoid this from happening again, we've become specialists at creating strategies that promote backtest accuracy. We work within the limits of the simulation. One of those limits is historical bar calculation; the other is the historical data itself. We can manage the former by switching to minute based data over range, but the latter is somewhat out of our control. I've tried several data providers and the issue is the same -- audits of longer dated backtest results tend to fail. This is not the case with market replay data, but it is difficult to run strategies in market replay for 6 months, let alone 5 years.
Finally, switching from reasons based on practical application to personal theory, I tend to value the most recent performance of a strategy over its performance two years ago. It's more relevant. I'm also more interested in the degree to which the profit factor changes when we update our performance charts every two months than I am in what the profit factor was 5 years ago. This is why we update the performance of strategies every two months.
why do you provide 1 year backtests only?
i would like to see how the algos performed in previous years and in different market conditions
It's a fair question. And ultimately, I made the final decision on this one.
The short answer is: If I trusted the data more, I would, but I don't.
The long answer is: Knowing that a strategy performed well for the last 5 years as opposed to the last year would certainly make me 'feel' better about trading it live, but just because you run a 5 year backtest doesn't mean you're going to get the answer to that question. One of our primary concerns is backtest risk or the accuracy of the standard backtest. Standard backtests use historical data, which is inherently flawed and I believe long dated backtests can exacerbate the issue. This is because the data tends to deteriorate the further back you go. Even if the strategy performs better in the past than it does in the current year, I wouldn't trust it. Unfortunately, we've been impaled by a few of our own unicorns already, which is to say, we know how inaccurate a backtest can be (If you follow the newsletter, you know what I'm talking about). To avoid this from happening again, we've become specialists at creating strategies that promote backtest accuracy. We work within the limits of the simulation. One of those limits is historical bar calculation; the other is the historical data itself. We can manage the former by switching to minute based data over range, but the latter is somewhat out of our control. I've tried several data providers and the issue is the same -- audits of longer dated backtest results tend to fail. This is not the case with market replay data, but it is difficult to run strategies in market replay for 6 months, let alone 5 years.
Finally, switching from reasons based on practical application to personal theory, I tend to value the most recent performance of a strategy over its performance two years ago. It's more relevant. I'm also more interested in the degree to which the profit factor changes when we update our performance charts every two months than I am in what the profit factor was 5 years ago. This is why we update the performance of strategies every two months.