2024 Q4 Forward Test Update
And they're off! The 2024 Q4 Forward Test Begins In Less Than 5 Minutes...
Important: There is no guarantee that ATS strategies will have the same performance in the future. I use backtests and forward tests to compare historical strategy performance. Backtests are based on historical data, not real-time data so the results shared are hypothetical, not real. Forward tests are based on live data, however, they use a simulated account. Any success I have with live trading is untypical. Trading futures is extremely risky. You should only use risk capital to fund live futures accounts and if you do trade live, be prepared to lose your entire account. There are no guarantees that any performance you see here will continue in the future. I recommend using ATS strategies in simulated trading until you/we find the holy grail of trade strategy.
"The essence of strategy is choosing what not to do."
— Michael E. Porter
Every few months I like to climb the highest tree in search of a broader view of the HUNT. In particular, I like to review:
where we’ve come from,
where we are now; and,
where we’re going to from here.
I am laser focused on one end — finding the holy grail of automated trading strategy. I have at times veered from this goal in search of adjacent pursuits, but I use posts like this to refocus and re-align myself to that original goal.
So, how far have we come?
I think we’ve come a long way. Here’s a list of past updates to help answer that question:
ATS also published the following posts/strategies/research in Q3 of 2024:
We also completed the first Live Test in July and I plan on starting a second Live Test in January of 2025 based on what’s learned from running the Q4 portfolio.
In the way of Renaissance Technologies, the question we’re asking isn’t ‘why does something work’, but ‘what works’. That’s what Q4 is all about. The Q4 Forward Test Portfolio is a showcase of what worked in Q1, Q2 and Q3 of 2024.
Q4 2024 - Where Are We Going?
"Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win."
— Sun Tzu
Before talking about where we are today in the hunt, I want to jump ahead to talk about where we’re going, because something big has happened.
In case you missed it, we’re on the hunt for the holy grail of automated trade strategy. We haven’t found the holy grail yet, but I’ve found many gems along the way and they culminate in the 2024 Q4 Forward Test Portfolio that I’m going to share with you today.
I’ve also met some gems in the form of like-minded people on the same journey. I am truly grateful for the opportunity to share this information with you at this particular time in AI history.
I believe we’ve entered the golden age of automated trading strategy creation (among other things). As a ‘non-coder’, being able to use LLMs to create the code for a strategy I relate in natural human language feels like magic. Can you imagine what Jesse Livermore, Richard Wyckoff or Munehisa Honma would have done with OpenAI o1? And it keeps getting better every day. At this rate, I can hardly imagine where we’ll be in six months, let alone a year. So please bare with me as I try to navigate these crazy times. One thing is certain, if you aren’t using these tools to enhance your hunt (whatever that hunt may be), you’re losing ground.
Due to numerous limitations, we’ve been diligent, but the hunt has been slow. The animal we’re tracking is capable of speeds well beyond our own, but now we’ve been given something like a magic carpet. It’s hard to put into words, but I know exactly how the turtle feels in this 10 second video:
As a newsletter designed for non-coding traders, I’ve used NT8’s Strategy Builder as a way to create and communicate automated trading strategies with you. It has been an extraordinary tool, but today we have something much better in LLMs.
I don’t know if this new generation of strategies will contain the holy grail, but they are strategies I’ve been wanting to create for a long time. I’ve been creating one or two a day. They use nuanced risk management structures, dynamic position sizing logic, and entry/exit options that I could only dream about before. I’ll be adding these strategies to an Incubator Portfolio. I hope to share some of the best performing systems with you throughout the quarter.
What does this mean for strategy development going forward? It means that strategies will no longer be created with the Strategy Builder, which means you won’t be able to edit strategies in the Strategy Builder, but you can still use all strategies on NT8. If anyone finds this challenging, please let me know and I’ll walk you through it.
I discuss how I used LLMs to create Strategy 85 here. You can also use LLMs to help categorize, label and identify ATS strategies. I am often asked if ATS strategies are available for use on other platforms. Two months ago, my answer was “no”. Today, my answer is,
“Yes. Copy and paste the code of any ATS strategy into an LLM and ask it to translate into the coding language of the platform you use”.
You can also use LLMs to build your own strategy, and/or ask questions about any strategy by uploading the code. You can ask it to improve on the strategy based on a certain level of fitness (more trades or a higher profit factor). You can give it your manual trade data and ask for an objective assessment. I’m also looking forward to seeing how it can help with portfolio strategy and I’ll share any revelations made with you in future Mudder Reports.
I’m also having fun learning how to apply the applications that use LLMs for our hunt. For example Google Notebook LM has the ability to upload several files (like Claude Project) to use as source material for the LLM. One of my favorite features is the ability to create a podcast between two people discussing the work, which is much easier to digest for me. I plan on seeing what it can do for strategy, data series or charts as source files this week.
So that’s where I see us going. Let’s get clear on where we are today?
Where Are We Today: The 2024 Q4 Forward Test Portfolio
Even without the help of LLMs, we are looking at what promises to be the best learning quarter of the hunt so far.
Q3 was the first quarter that I started tracking the performance of each variation in the forward test.
What was I doing before?
Well, there were over 150 strategy variations in the Q2 and Q1 forward test, but only 40 or so strategies were actually being run.
Just to make sure we’re on the same page, a “strategy” is the command structure, but that command structure has many different “variations”. Those variations can pertain to any instrument for any chart. It may involve a different set of parameters or time filters. It may calculate on bar close or on each tick. It may have dynamic position sizing or trade the same size every trade. Each of these variations can be applied to the same strategy. Those strategies that perform well across variations are considered stronger or more robust.
Prior to Q3 of 2024, each strategy in the portfolio had its own account and all strategy variations rolled up to the same strategy account. There are many flaws to this methodology. I’m almost embarrassed to admit that I did it this way for so long, but this newsletter is about holding myself accountable as much as it is documentation. I hope you can learn from my errors along the way.
So in Q3 I created a separate sim account for each strategy variation which was tedious, but well worth the effort. You can view the Q3 Forward Test Portfolio at the bottom of the Strategy Description page.
I also did something else in Q3 that was radical — I abandoned the backtest to increase robustness.
I like the definition above from MultiCharts, but in our case we’re using forward tests to analyze robustness instead of backtests. Likewise, instead of creating a forward test based on backtests, I created a forward test based on the results of the prior forward test. Again, I feel like a fool for not doing this sooner and an even bigger one for having to admit the oversight here, but the goal is to document, reflect and improve.
For anyone that’s new to ATS, the difference between a backtest and a forward test is that a forward test does not rely on historical data, which is riddled with errors (in both the data itself and the simulation of that data). Instead, I’m running all strategies listed in the Q4 Forward Test on simulated accounts using live data. It’s not the same as trading on a live account, but it’s better than using historical data because it’s one step closer to reality. The only thing that we can’t simulate is the order matching algorithm of the broker/dealer, which is generally proprietary.
All the strategies in the Q4 Forward Test are different. Some include a risk management plan; some only make long trades; some should only be traded with metal futures; while others should only be traded with equities. There are scalping strategies, high win rate strategies, time based strategies, momentum strategies and even AI generated strategies.
Here’s a look at the strategies in the Q4 Forward Test Portfolio. Subscribers, scroll to the bottom for a link to the full Forward Test with Parameter Settings for each variation.
The first column is the Strategy, the second column is the instrument, and third is the data series. These columns are followed by net profit, gross profit, gross loss and profit factor based on Q3 forward test data.
In total, without taking new strategies into consideration, the Q4 Forward Test Portfolio made $249,337 in net profit in Q3, with a gross profit of $383,013 and gross loss of -$114,954, resulting in an overall profit factor of 3.33. I expect the net profit for Q4 to be higher, but less volatile for the following reasons:
most highly volatile strategies, especially those with “looser” command systems, blew up in Q3 because the market was highly volatile. These strategies have been eliminated from the Q4 Forward Test Portfolio.
there are 14 new strategy variations based on those variations that performed well in Q3.
sizing logic is based on the strength or robustness of the strategy.
Let’s get into the details.