July Update: In The Last 12 months Our Strategies Made Over $2.2M
We've Officially Transitioned From NT7 to NT8...The Hunt Continues
Every two months we like to take a look at how our top automated trading strategies are performing and update you on our key findings, highlights, takeaways and what’s in the pipeline.
What’s different about this update is that it’s the first time we’re updating you on strategy performance using Ninjatrader 8 (NT8) to run backtests — this is because NT8 is much better than Ninjatrader 7 (NT7) for accuracy.
For the sake of clarity, we’re going to compare the difference between Ninjatrader 8 and Ninjatrader 7 in both backtest calculation and performance.
For the sake of consistency, we’re going to show both an update with NT7 data as well as NT8 data.
So the goal of this update is twofold:
We want to see how well our strategies are performing by comparing them to the last update (NT7). Purpose: To see if our strategies are consistently performing well.
We want to compare NT7 to NT8: Purpose: To ensure that our backtests are still performing well using more accurate backtest methods. We know that NT8 is more accurate than NT7 because backtest data is calculated on a more granular level in NT8, so we expect to see a difference between performance results, but we don’t know how severe that difference will be or if it will be positive or negative.
Ultimately, the headline is that we’ve established a new baseline.
Why Switch To NT8
We’ve been doing a lot of research over the last 3 weeks on backtest methodologies. As a result, we’ve implemented several changes to our process that will increase accuracy, but the most important concern was the way in which NT7 calculates historical data.
As a primer, it takes a lot of data to mimic all the movements of the market, so backtest software creates methodologies that try to mimic the market without using all the data required to do so. The result: holes in your data. These holes often lead to inaccurate backtest results, especially for bar charts that require more robust calculations. We witnessed these holes first hand when creating backtests on Renko charts in NT7. You can read more about that sad event here.
Ninjatrader was very aware that backtest accuracy was an issue with their software (and with backtest software in general) so they made several changes to the backtest methodology in NT8. Originally we were going to move to NT8 at the beginning of 2021, but the discrepancy between backtest results and market replay data created an urgency. There’s no point in hunting for the holy grail of trade strategy if our maps are leading us down the wrong path.
Unlike NT7, NT8 provides two options to control the granularity of historical order fill processing: Standard and High.
The Standard order fill resolution (the fastest) breaks each historical bar into three virtual bars to mimic the movement of price within each bar's time-frame, which provides more realistic historical data compared to traditional backtesting calculation methods (used in NT7) that only use the open/high/low/close (OHLC) prices of the bar.
The three virtual bars are created based on the following logic:
1. Open price to the High price
2. High price to the Low price
3. Low price to the Close price
Here’s a visual example provided by Ninjatrader:
What happens if the open price of the bar is closer to the low price than the high price?
1. Open price to the Low price
2. Low price to the High price
3. High price to the Close price
What we just reviewed is the Standard or faster backtest methodology in NT8.
NT8 also allows for a High order fill resolution, which allows us to enter a secondary bar series to be used as the price data to fill orders. This provides us with more granular data (such as 1 tick or 1 second) so that the strategy or indicator contains the individual ticks/seconds in the backtest/historical data.
So, NT8 provides more accurate backtest results than NT7 at both the Standard and High order fill resolution level. That said, all of our backtests will be at the high order fill resolution level.
Now, let’s get into the update. We’re going to start with an NT7 update and then show the NT8 update for comparison.
Key takeaways from NT7: 1 Year Backtest Update
While we know NT8 provides more accurate results, and we’re going to provide those results next, we also think it’s important to compare apples to apples in the transition from NT7 to NT8. The following analysis is comparing NT7 update results only.
All strategies are based on the NQ futures contract.
March 6 backtest results:
(click here for the full March 6 update)
July 2 backtest results:
Let’s talk about the changes first.
we reduced the number of days traded from 365 to 253, which is the actual number of traded days in the year. This has the effect of artificially pushing up profit per day.
due to the potential for overfitting, we’ve highlighted all optimized strategies in yellow. These are the strategies that are most prone to overfitting and therefore may not be as consistent over time. To read more about overfitting, please see the following posts: Overfitting: What is it and what can we do about it and What Are We Doing To Ensure Backtest Accuracy?
thanks to a research question asked by a subscriber (how do we know how much we need to trade each strategy?) we’ve added a dollar value for max drawdown along with a measure of return (return on max drawdown), which is calculated by dividing net profit by the max drawdown. In this way, max drawdown is considered the max capital investment. You can use the dollar value of max drawdown as a proxy for how much capital you need to trade the strategy. And, the higher the return on max drawdown, the better the strategy is in terms of risk/reward. That said, only profit factor can provide an indication of the required reinvestment (profit factor is the ratio of gross loss and gross profit). You can read more about why we use profit factor to compare strategies here.
ideally, we’re looking for something with a low risk/reward ratio (characterized by a high return on max drawdown) and a low reinvestment requirement (characterized by a high profit factor).
As a side note: Matt, we heard you and looked into your suggestion, but decided to stay with profit factor rather than average profit and loss ratio. Yes, a few large winners/losers can sway the metric, but we want to factor in those extremes (at least for now). That said, we truly appreciate the suggestion.
Analysis of NT7 results:
In general, when comparing only the first 15 strategies: max drawdown decreased, percent profitable increased, trades per day decreased, but profitability and profit factor remained the same.
This general description can be applied to all strategies except: Strategies 1, 2, 5, 6, 10 and 11. These strategies had a slight increase in profitability. Net profitability for all 15 strategies increased just slightly to $1,062,870 from $1,040,285.
Why is this happening? The first quarter of 2020 was extremely volatile due to COVID.
In addition to a lower drawdown, our strategies also made fewer trades, which makes sense due to lower volatility.
In terms of the percentage of profitable trades, Strategy #8 remains the highest at 58%. Strategy #8 also had the third highest profit factor, with Strategy 15 rising to second place at 1.16, and Strategy #7 remaining in first place at 1.25.
Now that we’ve compared results for NT7, let’s see if NT8 provides the same results.
Key takeaways from NT8 Results: 1 Year Backtest Update
We knew NT8 would be more accurate, but we didn’t know it would show that our strategies were better than we thought.
1 year backtest results (NT7)
1 year backtest results (NT8)
The average number of trades is roughly the same, but average profitability increased from $55,739 (NT7) to $97,469 (NT8). While the average profit factor is lower, that is primarily due to an outlier in NT7 data (Strategy 19). In fact, if you only compare the first 15 strategies, the average profit factor increases from 1.07 to 1.12. Additionally, average max drawdown is considerably lower and return on max drawdown is higher. For the year, net profit increased from $1,226,255 (NT7) to $2,223,095 (NT8), with a total max drawdown of $416,665 (NT8). It is important to note that this increase is due almost entirely to the change from NT7 to NT8, not the strategies themselves. In layman’s terms, this means you could have made $2.2 million with a max investment of ~$420K.
Strategy 8 continues to have the highest number of profitable trades at 57%, but it also has one of the lowest profit factors. Strategy 4 was a surprise — the percent of profitable trades increased from 37.5% to 53.1%. Strategy 4 also had a profit factor of only 1.04 in NT7, but under NT8 the profit factor has increased to 1.20, which jives with our real-time performance. Strategy 7 continues to have one of the highest profit factors at 1.30.
Not surprisingly, optimized strategies did not hold up well in the update, however, Strategy 18 has the highest profit factor at 1.90. Again, optimized strategies (highlighted in yellow) are more prone to overfitting and are therefore not as consistent. We advise greater caution here.
Since the NT7 update saw no change in total profitability across all strategies, it is safe to assume that the increase in profitability from NT7 to NT8 is due almost entirely to the change in backtest methodology, which is more accurate than NT7. We run our strategies every day, so this validated our assumptions, but we had concerns when we backtested Renko bars instead of Range bars. Renko bars require more data than Range bars to calculate, so it is no surprise that NT7 was unable to handle the calculation. When we ran our strategies through NT8 using Renko bars, the results were much more in line with what we were seeing in real-time.
You may have noticed a new strategy added to the table - Strategy 23. While it has a low annual profit and only 241 trades for the year, it has the highest profit factor of all strategies that have not been optimized. Everyone on our email list will receive the analysis of Strategy 23, but only paid subscribers will receive instructions on how to duplicate it. That email will go out on Friday (7/23).
Highlights / Conclusion:
We continue to like Strategies 4, 7 and 8. We are also expecting great things from our new Strategy 23 and will continue to mine these strategies to see if we’re missing something.
While these are our favorite strategies, we’re still not happy. As a quick reminder, our goal is to find the holy grail of automated trade strategy, which we’ve defined as:
Profit factor greater than 3
Annual drawdown less than 3%
Annual return greater than 500%
Minimum daily net profit of -$1,000
Avg Daily profit greater than $1,000
Less than 5,000 trades annually
We have yet to find this illusive trade strategy, but we’re on the hunt!
If you subscribe to our newsletter, you’ll be among the first to find out when we do.
What’s in the pipeline:
So we’ve established a new baseline. This will be the last time we provide any backtest data ran on NT7.
We’re going to republish data on optimized strategies (16-22) using NT8s optimization options. You can expect the full publication on September 6 (subscribers only). We will be using three methods:
Walk Forward optimization,
Multi-Objective optimization, and
Monte Carlo Simulation
There are also multiple research requests in the works. Please see here for the full update on each research request. A clipped version of this update is below.
1) Do your strategies work on cryptocurrencies?
This is perhaps the most common question we get asked. We’re shooting for an August 21 (email list) publication to answer the question.
2) What are you doing about overfitting?
Outside of cryptocurrency, this is the most commonly asked question we get. Please see the following posts: Overfitting: What is it and what can we do about it and What Are We Doing To Ensure Backtest Accuracy?
3) What is the difference between Total Investment & Reinvestment:
About three months ago we received a question regarding total investment. This is our answer: Reinvestment Vs Total Investment.
4) Do your strategies work with Renko?
Due to the nature of Renko bar calculation, Renko backtests can only be done using market replay (publication date: tbd). Subscribers can click here for the full post.
5) Are certain trading days/periods better for automated strategies (earnings season, economic data releases, FOMC announcements, triple witching hour). If so, can we take advantage of these days in the future?
One thing we can count on is higher volatility and volume on certain days of the year. Does it also follow that our strategies operate better on these days? We’re hoping to have some definitive research on this by September 13 (subscribers only).
6) Can we develop a strategy that does better in the morning volatility/lunchtime lull?
We’ll be publishing this research on August 29 (subscribers only).
7) Can you create a correlation heat map for your strategies?
Publication date: tbd
8) What is slippage and how are we accounting for it?
Slippage is anything that contributes to receiving a different price than you wanted on the trade execution. In other words, the trader receives a different execution price than he wanted. It can be positive or negative. Everyone is aware of negative slippage, but few consider positive slippage, which is just as common. Positive slippage is when the price you receive is better than the one you were attempting to buy/sell at.
One of the main ways to avoid slippage is to make use of limit orders instead of market orders. This is because a limit order should only be filled at your desired price.
9) Is there a way to discern a choppy, high volume session from a normal, high volume session?
The main sign for me is longer tails. The longer the tail, the choppier the water. The question is, how can we take advantage of this; that is, is there a strategy that does well in a choppy session?
I reached out to a friend that surfs for help with this. This may sound odd, but the market moves in waves and I’ve found that surfers have a natural proclivity for trading. He said the best thing to do in choppy waves is referred to as “tacking”. He described it as the same movement you use when moving a skateboard - back and fourth movements. Indecently, this is the same thing you do when you’re on a sunfish boat and you’re sailing against the wind. This translates into the use of limit orders, stops and take profits. So instead of using a strategy that’s looking for long waves or big price movements, you’ll need a strategy that uses limit orders to get in, longer stops and shorter take profits. The hard part is creating a program that recognizes a choppy session.
Publication date: August 21 (subscribers only)
10) Can you teach me how to trade?
No, but fret not. None of us want to teach at the moment, but before you do anything else, read this: Trading Is A Dark And Scary Forest. This tells you a little about my personal journey and what you can expect as a trader. Then you may want to read this: We Are Treasure Hunters Searching For The Holy Grail Of Automated Trade Strategy to find out about our goal at ATS. And finally, if you’re new to trading, we highly recommend simulated trading for at least a year.
I also want to suggest doing a search for the company TopStep online. Review all of their videos on Youtube. Use their Youtube channel as a place to learn. You don’t need to buy anything. Just watch the morning call, review all of their videos and sign up for their blog. I believe they also have a free group class on Wednesdays given by Hoag — don’t take any classes unless they’re with Hoag or Eddie.
Treat your search for a teacher with respect. You are looking for a Shihan — someone that can teach you how to learn more from losing than winning. Expect some hazing. This could be one of the most important decisions of your life if you do it right.
11) What’s your favorite platform?
Our favorite platform is NinjaTrader, though we are trying out several others. If you don’t know how to use Ninjatrader, they also have some great tutorials and live events for free. They also have free data from Kenetic. The data is not live (I believe there’s a one day delay), but you can practice our strategies on it. Again, you don’t have to buy anything.
12) What can we learn from strategies with a profit factor of 1?
We started this research with the post: Can We Maximize The Profit Factor By Focusing On Timing?
We found that timing can increase the profit factor by 15-20% on our strategies and we wanted to continue the research.
Specifically, we wanted to know:
If profit margin on all strategies tends to trend toward 0 or a profit factor of 1, is there a way to capitalize on this truth?
Is it possible to achieve a higher profit factor using strategies based on reversion to the mean theory?
Follow up: we looked at Bollinger Bands, but had no success. Currently we are looking at divergence theory, however, we are finding the most success with strategies that “flip” under certain conditions. In other words, if the strategy has a profit factor of 1 it means 50% of the time it’s right and 50% of the time it’s wrong (in general). Is it possible to identify the conditional that flips the trade signal? This is a hard one, but we’re working on it. Again, this is our most promising strategy idea, but we don’t expect to publish any research on it until September.
Publication date: September 2 (subscribers only).
13) What can market physics teach us about trading strategy?
Publication date: August 15 (email list)
14) Can you create strategies that only trade during certain conditions?
This is what we do. Conditions includes timing, volume, choppiness of the market.
We will publish a definitive answer to this question, however, on August 2 (subscribers only).
15) Review of Multicharts: Publication date: October 2 (email list).
To all paid subscribers: If you don’t see your question here and haven’t received an answer to your question, please respond to this post or email us directly at automatedtradingstrategies@substack.com.
Click here for links to all strategies.