Automated Trading Strategies: July 2022 Portfolio Update
Over the last 12 months our top strategies made over $2.8M based on NT8 backtest results. Average profit factor increased from 1.40 to 2.02 since the May update.
Hi Traders,
In case you didn’t know, we’re on the hunt for the holy grail of automated trade strategy. We define the holy grail of trade strategy as having the following attributes:
Profit factor (gross profit/gross loss) greater than 3
Annual max drawdown less than 3%
Annual return greater than 500%
Maximum daily low of -$1,000
Avg Daily profit greater than $1,000
Less than 5,000 trades annually
Greater than 253 trades annually
Every two months I like to look at how our top automated trading strategies are performing and update you on key findings, highlights, takeaways and what’s in the pipeline. This is our July 2022 update. You can view past updates here:
It’s been well over a year since we started this. With every update, the data grows richer and our observations more insightful. We’ve learned a lot and we use we what we learn in each successive strategy.
In our May update I talked about volatility and its impact on automated trading strategies. What we found is that like everything else, the primary impact and therefore vulnerability on automated trading strategies relates to the backtest. In particular, it increases the likelihood that backtest results will be misleading. We refer to this as backtest risk.
Now I want you to think about what it means to have a high backtest risk. Ultimately, it means it will take much longer for you to find the holy grail of automated trade strategy than it would if your backtest was always 100% accurate.
What exactly is backtest risk? It’s the risk that the backtest will be different from the actual live performance. It is the fear of every trader on the hunt for the holy grail. If you aren’t afraid, you should be. It is by far our biggest enemy. While volatility increases net income, it also amplifies the power of this enemy. Not only does it fool you into believing you’ve found gold, but it wastes precious time. This is what makes the hunt so long and arduous. It’s not strategy development, but backtest risk or put another way, the inadequacy of the simulation.
What can we do to defend against this?
In addition to conducting a standard backtest, we can conduct a high-order fill backtest (at the tick level), followed by a market replay backtest, followed by a live simulated test. Each test requires more time, but each test improves the accuracy of results.
We can use minute based data rather than range or Renko based data. Range and Renko are more complex bar formations so the simulation has a hard time recreating the bar formation for the backtest. Switching to minute based data doesn’t improve the simulation, but it improves backtest accuracy by focusing on what the simulation does best.
We can use more than one asset in our backtest/portfolio. Using more than one futures contract increases diversification and decreases risk. It also allows us to expand the hunt to the highest performing assets.
We then tried to apply these lessons to the hunt.
How did we do this?
We reran all strategies for comparison to see how they performed with a 1, 5, 10, 15, 30, 45, 60 and 120 minute data series on the following futures contracts:
6B - British Pound (FX)
6E - Euro (FX)
CL - Crude Oil
ES - E-Mini S&P 500
FDAX - DAX
GC - Gold
NQ - E-Mini NASDAQ 100
RTY - E-Mini Russell 2000
YM - Mini Dow
ZB - 30 year Bond
ZN - 10Year Note
ZS - Soybeans
ZW - Wheat
We then selected the best performing strategies. These were the results:
You will notice that not all strategies are included on this performance chart; only the strategies and contracts that performed well using a minute based data series.
We developed this new portfolio by optimizing strategies based on the highest performing contract type and data series. I’m not a big fan of optimization because it tends to create strategies that are over-fitted and unreliable. That is, over-fitted strategies, especially those that have been optimized based on indicator parameters, have a high rate of alpha decay. Naturally, my biggest concern for this new portfolio is that it won’t perform well over time. The July update will be the first test.
So what does the July update tell us?
We’re on the right track! The July update suggests that optimizing based on data series does not hurt long term performance. In fact, in some cases the strategy performed even better. These are the results of the July update:
Over the last 12 months our top strategies made over $2.8M based on NT8 backtest results. Additionally, average profit factor increased from 1.40 to 2.02.
Which strategies performed the best?
Strategy 2 (also available in the ATS Mini Newsletter) performed well.
Strategy 2 (5 minute NQ) - made 322 trades on a profit factor of 1.64. That’s a slight deterioration from 1.68 in the May update, but income is also higher.
Strategy 3 (also available in the ATS Mini Newsletter) has always been one of my favorite strategies, but the number of trades it takes usually makes it prohibitive. However, once we increased the data series to 120 minutes (rather than 36 range) the trade count became easier to manage.
Strategy 3 (120 minute NQ) - made 669 trades over the last twelve months, had a profit factor of 1.38 (up from 1.33 in the May update), and made $193K
Strategy 3 (120 minute FDAX) - made 403 trades, had a profit factor of 1.26 and made $85K
Strategy 6 was a big surprise in the last update making $142K with a profit factor of 1.29. In the July update Strategy 6 made $185K on a 1.35 profit factor. So the performance of the strategy increased. (Note: Strategy 6 will be published to the ATS Mini Newsletter on July 15.)
Strategy 7 (FDAX) was also a surprise. It went from making $104K in our May update to $173K in our July update. Additionally, profit factor increased from 1.24 to 1.40.
Strategy 32 was one of our highest performing strategies on the May update.
Strategy 32 (45 minute, FDAX) had a profit factor of 1.72 with the May update, which improved to 1.79 in the July update
Strategy 32 (30 minute, ZB) had a profit factor of 1.81 with the May update, which decreased slightly to 1.53 in the July update
In the May update, none of our strategies had a profit factor over 1.81. In the July update both Strategies 48 and 50 have profit factor’s over 1.81:
Strategy 48 (1 minute, NQ) made $61K and had a profit factor of 3.01
Strategy 48 (1 minute, NQ, no exit on close) made $63K and had a profit factor of 3.08
If we look at Strategy 48 for use with a net income based risk management system that uses up to 5 contracts, net income grows considerably:
Strategy 48c (1 minute, NQ) made $288K and had a profit factor of 3.00
Strategy 48c (1 minute, NQ, no exit on close) made $292K and had a profit factor of 3.05
Strategy 48 is also the only strategy that makes over $1K per day on average. It also has the highest profit per trade at over $600.
Finally, we have Strategy 50, our newest strategy. In total, there are 8 different variations of Strategy 50, and they all make less than $20K for the year, but profit factors are the highest we’ve seen. So, we’ve combined the top performing variations into one portfolio. We’re also looking at a portfolio based on the highest performing contracts for Strategy 50:
Strategy 50 (10 minute, YM) only made 33 trades and $8K, with a profit factor of 11.57.
Strategy 50 (10 minute, ES) only made 35 trades and $13K, with a profit factor of 8.61.
Strategy 50 (10 minute, NQ) only made 33 trades and $16K, with a profit factor of 3.79.
Our current test looks at a portfolio of 9 contracts using Strategy 50 and its reverse.
We’ll keep all subscribers posted on updates in the Mudder Report.
I also want to point out Strategies 11, 15, and 50 for having the lowest daily net loss — all three never lost more than $1,000 on any given day of the year.
I also want to highlight Strategies 7, 12, and 14 — these strategies never went below zero. That is, cumulative net profit was always positive.
I also want to point out that Strategies 11, 15, 47, 48 and 50 have max drawdown’s less than 3%.
If you’re more interested in the percentage of profitable trades, we had some winners there as well:
94% of Strategy 44b’s trades were profitable
84% of Strategy 47’s (GC) trades were profitable
87% of Strategy 50’s (YM) trades were profitable
The July update did not disappoint. We’re hoping to see the same results on the September update, especially from Strategies 44 through 50.
Strategies Take Years To Develop
I’m a fan of The Algorithmic Trading Blog. The author is relatively new to trading, but has the uncanny ability to develop very insightful content about what it takes to develop an automated trading system. In the post, “10 Things to Learn From Professional Traders”, which would have been more aptly titled “10 Things Professional Traders Can Teach Us About Developing Automated Trading Strategies”, Chris Shields provides a literary review on some of the top automated trading strategies on the market. While I disagree with some of his insights, the first insight slapped me square in the face: Strategies take years to develop.
I’ve always been faster at figuring things out than most people (at least I like to think I am). This isn’t because I’m smarter, it’s because I’m neurotic. I obsess. When we started this hunt I thought we’d have a strategy developed in one year — that was 18 months ago.
So, when I read the first insight in Shield’s post, it was both a source of relief and a challenge. Relief in that those that have presumably been successful on the hunt took years to get there. It was also a challenge because as I read his blog post, what became clear to me was that the hardest part about finding that unicorn strategy is testing. While not implicitly touched upon in the post, it became clear that what takes time is testing the strategy, not the actual development. Any attempt to shorten the time it takes to reach the holy grail must involve improving backtest accuracy as much as it does strategy development.
If this is the case, why don’t we see more talk about improved backtest accuracy from the automated trading community? The truth is, we do and they tend to look like the following post from Capablanca in June of 2019:
This is perhaps the most common finding on Ninjatrader’s forum regarding backtest accuracy. That is, live results and market replay tend to yield the same performance, but the standard backtest from the strategy analyzer is often misleading.
And this was Ninjatrader’s response to Capablanca’s post:
To read more about Ninjatrader’s response to backrest accuracy read: Discrepancies: Real-Time vs Backtest.
So, while we traders are on the hunt for the holy grail of automated trade strategy, automated trading platforms like Ninjatrader are on the hunt for a backtest engine that is able to perfectly simulate the market. Now that we know how critical backtest accuracy is to the hunt, we must stay vigilant on finding a way to perfect it.
In a past life I worked in strategic procurement for Harvard University. It was my job to select the best vendor for the community. To build consensus from key buyers in the community I often held open vendor selection presentations. My job was to ask the question every vendor was trying to avoid — it was usually the question that mattered the most. Likewise, if I was tasked with finding the best automated trading platform for our subscribers I would ask the question: “How accurate are your backtests to live results?” This is the only question that really matters. All the beautiful charting software, amazing indicator sets, top notch support — it all means nothing if the backtest is inaccurate.
It should be no wonder that I’m currently running my own investigation. I’m sending emails to the best trading platforms out there and I’m only asking them one thing: “How accurate are your backtests to live results?” It will take a few months to synthesize my findings into something we can act on in the hunt, but paid subscribers of ATS and ATS Mini will be the first to know what I find.
Backtest Accuracy is More Important Than Strategy Development
I can’t point the finger at backtesting platforms without pointing the same finger at the traders selling their strategies on the web. At ATS, we don’t claim to have found the holy grail…yet. Our strategies are the best of what we’ve found via standard backtests, not live testing. While we’ve greatly improved backtest accuracy by switching to minute based data, we are still working on ways to close the gap.
We implemented a backtest risk score, which makes a prediction on how accurate each strategy’s backtest will be based on specific attributes within the strategy. That said, it’s just a prediction model. We need more.
So in addition to the backtest score, and in addition to researching other platforms, I’m also developing a way to document strategy performance using market replay and simulated live results. In my research, I have yet to see this done before.
To be clear, we’re comparing three types of backtest results:
Standard Backtest using historical data from the Strategy Analyzer. This data is lacking in data that would make perfect bar calculations. The Strategy Analyzer’s best attribute is speed, not accuracy.
Market Replay using a download of actual data used to create a simulated market as a visual replay. Market Replay’s best attribute is accuracy, not speed. It takes a great deal of time and effort to download one year of market replay data for a single instrument. Also, Ninjatrader only keeps this information available for 3 months. So you have to download the data yourself. Once the data is downloaded, it then takes about a day to run the strategy through market replay, even at maximum speed.
Live Simulated Testing using a virtual server located close to the exchange. In our case, Chicago is the home of our exchange so we’re using a virtual server in Chicago. ‘Live simulated testing'’s best attribute is accuracy, not speed. There is no way to speed up live simulated testing — it is what it is.
Backtests for 2 and 3 tend to be the same as actual live trading, but the Standard Backtest, while much faster, can be misleading — that’s the tradeoff to speed.
I’m currently working on a process to share/compare the results of all three backtests (standard, market replay, live simulated) at the same time. From my own research, this will be the first time backtest results are shared using all three results. I’ve scoured the Internet for something similar and I haven’t found anything. The truth is, and I’m not trying to call anyone out, but you don’t see traders talk about this much, especially if they’re trying to sell you a strategy. No doubt it breeds skepticism about the strategy’s performance results. At ATS we can’t afford to dance around this issue if we’re going to find the holy grail in less than 2 years, so we’re taking the challenge head on.
The goal is to compare all three backtest results over time. In this way, even if we can’t find a better platform, we can close the gap concurrently using NT8. The progress of these tests will be shared in the Mudder Report. We are currently testing:
Strategy 31
Strategy 32
Strategy 2
Strategy 14
Strategy 11
Strategy 15
Strategy 50 and it’s reverse
Strategy 50a1
Strategy 50c
Strategy 50ft
Strategy 50i
Strategy 50d
Strategy 0
Strategy 40
Strategy 49
We’re also testing Strategy 51, our next strategy to be published on July 15. I’m also planning on using this framework as a way to test the accuracy of other trading platforms.
Housekeeping
One thing many of you have asked for is a less expensive version of this newsletter. One of our free subscribers suggested a kind of mini version of ATS; one that included a drip of our older strategies. I think it’s a great idea. So in commemoration of our 50th strategy, we launched the ATS Mini Newsletter. It will start with strategies 1 through 5.
Like ATS, subscribers will receive a new strategy every two weeks. For example, on July 15, ATS subscribers will receive Strategy 51; and ATS Mini subscribers will receive Strategy 6. The ATS Mini Newsletter is the starter class; The ATS Newsletter is the master class. The ATS Mini subscription price is $29/month. The ATS subscription price is $1,249/month.
To subscribe to the ATS Mini Newsletter click here.
To subscribe to the original ATS Newsletter click on the Subscribe button below.
I also want to make a request of the community. If you can recommend or point us toward any platforms/traders that are addressing the issue of backtest accuracy for automated trading systems, please let me know.
What’s up next?
Strategy 51 (coming out July 15) is a build up of Strategy 50 — we’ve isolated YM and ES (the highest profit factor variations for Strategy 50) and added a risk management system like the one we placed on Strategy 48. Preliminary cash flows for this strategy are very interesting. Strategy 52 (Coming out August 1) is our first strategy based on the use of Bollinger Bands. Preliminary tests are showing an annual profit factor of ~2.58.
We’re also looking into creating strategies around BIT or Nano Bitcoin futures. This new futures contract just started trading so the market isn’t there yet, but I believe it will develop very quickly. Not only is the contract being offered with commission-free trading at 1/100th the size of one Bitcoin, but BIT is being traded on a regulated market that’s as easy to short as it is to buy. Suddenly, a strategy that makes a large number of trades (like Strategy 3) is viable. BIT is traded on the Coinbase Derivatives Exchange, a CFTC regulated Designated Contract Markets (DCM) futures exchange. This is its first listed crypto derivatives product. To learn more about the benefits of BIT, click here.
As always, if you have any questions/comments/suggestions, please let me know by replying to this post.
Trade well,
Celan, founder of ATS
why do you provide 1 year backtests only?
i would like to see how the algos performed in previous years and in different market conditions