First Mudder Report of 2024: ATS Forward Test Update

Strategy 57 made $68K yesterday.

Important: There is no guarantee that these strategies will have the same performance in the future. I use backtests to compare historical strategy performance. Backtests are based on historical data, not real-time data so the results shared are hypothetical, not real. There are no guarantees that this performance will continue in the future. Trading futures is extremely risky. If you trade futures live, be prepared to lose your entire account. I recommend using these strategies in simulated trading until you/we find the holy grail of trade strategy.

---

We haven’t found the holy grail of automated trade strategy yet, but we get closer with every strategy. Click here for links to all strategy descriptions and the most recent performance chart from the backtest.

Edouard Manet 053, The Races at Longchamp, 1864, Art Institute of Chicago

Backtests are what they are. Some are better than others. For example, backtests based on: high frequency bar formations; minute and tick based data series, and; the close of the previous bar as a data point, are generally more reliable. But that’s not what this post is about.

This post is about the results of the ATS forward test. The ATS forward test was started in January of 2023 because we realized that backtests are largely unreliable unless you know how to stack the deck in your favor by using some of the observations I provided above.

If you’re a trader on the hunt, you know the struggle. Here’s a quote from the B/O Trading Blog on a post that came out just this morning:

I could NOT agree more. You do all this work and analysis and at the end of the day, it’s all just a bunch of mental masturbation.

Photo by Justin Luebke on Unsplash

So you have two options when you come to this realization. You can spend time trying to make your backtests more accurate by 1) improving your data and 2) finding a better market simulation. Both of these will most certainly improve the accuracy of your backtests. In the world of algo and automated trading, this is its own little holy grail—the ability to create backtests that are 100% accurate. If you have a platform that can do this, please let me know.

The other option you have is to focus on forward tests. Unlike backtests, the payoff is not instant. You have to earn the data, but like most things with any value, it’s worth the wait.

And, while it looks like we’re going to be migrating to another platform as well, we’re also putting more focus on the forward test. This is hard, I know. That’s why you rarely see it. Anyone can show you a backtest, but very few are willing to create a forward test and those that do turn into a trader version of Gollum.

I get it. That’s what I want to do as well, but I can also tell you that sharing the data has its advantages. I would say they outweigh the disadvantages, but that’s a discussion for a future post. The focus of this post is on forward test results.

What’s changed in the forward test from Q4 of 2023 to Q1 of 2024?

A lot. We went from modeling out this backtested portfolio, to this one. To be clear, these are backtests that are used to create the forward test. Changes were made based on lessons learned from the forward test as well the addition of new strategies that are designed to backtest with more accurate results.

In the forward test results I’m about to show you, you’ll see that some strategies did well and others did not. What tends to happen is that performance ebbs and flows through the year. Just like anything else, these strategies appear to move in cycles. I’ll talk a bit more about these cycles in the upcoming Live Test Update. The live test is even better than the forward test, but you have to start somewhere. What you don’t want to do is go straight from the backtest to the live test. If I were to rank the ability of testing methods on the market simulation it would be:

1. Live Forward Test (KING - live data on a live account)

2. Simulated Forward Test (live data on a simulated account)

3. Market replay Simulation (backtested on historical data)

4. High-order resolution Simulation (backtested on historical data)

5. Standard Simulation (backtested on historical data)

For more detail on each of these tests, please see this previous Mudder Report.

In general, the more reliable the test, the more time it takes to run and ‘there’s the rub’, as Shakespeare would say. In other words, accuracy takes time. So there’s no way to get around it—the best way to test strategies before going live is with a forward test, using live data on a simulated account. I am fairly certain that one of the main reasons people lose so much money on the hunt is because they skip this critical step. Here’s a post I recently published on how to do it:

How To Set Up Your Own Forward Test

ATS was started three years ago, but the ATS forward test didn’t start until a year ago. I regret that. It is the single most important tool in our arsenal. A great way to learn from my error is to start your own forward test today. I started with only a few strategies and then continued to add more throughout the year.

Once we get the forward test results, then what? Well, we can see if the strategies really are profitable or if the backtest was faulty in some way.

Most of you saw the results of the forward test in the last Mudder Report, but the question I keep getting from subscribers is, “What about profit factor?” It’s a good question. The following forward test results include profit factor as well.

Before showing you the best performing strategies in the ATS 2023 Forward Test, I want to follow up with survey answers from the last Mudder Report. The first question was: How should we handle updates to the data series (frequency of bar formation) for each strategy?

• 44% of you said that you want to continue the update every 3 months

• 25% of you said that we should never update after the first optimization; and,

• 19% of you said we should just use 60 minutes for all backtests.

All make sense to me. There are advantages to each. We will be continuing the update every 3 months.

Here’s a second question I asked that received more votes than the first: Do you think sharing strategies hurts or helps our mission? This is a bit like a chicken asking the man with the knife if it’s better to run away fast or slow, but I’m counting on your honesty.

• 53% of you said it helps

• 26% said both, but helps more than hurts

• 21% said it hurts.

Thank you for your input. I will take it into consideration over the coming months. I think the truth is likely somewhere in the middle. The trick is carving out those parts that hurt.

I have one more question for you:

If you have another idea, send it to me.

In addition to reviewing the forward test results, I’m also going to give you an update on the election day strategy: Strategy 73.

Now let’s get into the forward test results.

The Forward Test Is Evolving

These are the results of the forward test. The trades are being made on a simulated account, so the results are still hypothetical, but the data driving the simulation is live. Note, this does not cover all weeks of the year. Also note that strategy variations vary by instrument, data series, parameter setting and contract size. The charts below provide the: 1) strategy, 2) net profit, 3) start date (not all strategies were started at the same time), 4) trade count; and, 5) profit factor for all strategies in the forward test. I’ll also provide a quick overview of which strategies performed the best in the first two weeks of 2024.

The purpose of the Mudder Report 3.0 was to track real-time performance, not to maximize performance. In 2024, however, I’ve decided to focus on performance. This is why you will see that some strategies are not in the Q1 2024 backtest. This is also why some strategies contain more than one variation. The hope is to gain some understanding around the profitability of different configurations. This is especially salient for strategies with a high win rate and small trade count. A great example of this is Strategy 57, which the Q1 Backtested portfolio shows only made ~30 trades for the year on any given variation. As you’ll see from the chart, Strategy 57 actually has 6 different variations and in order to increase the trade count, I’m also trading 3 contracts on each leg. In other words, if all variations trigger, we’re trading 18 contracts.

That’s exactly what happened yesterday when Poor Powell came out and did what he’s done for the last 2 years—thrown a wet blanket on the party—causing Strategy 57 to trigger and break all Mudder report records to date for best performance in a single day at $68,680. No, I can’t blame that on Powell—that was a technical downturn and Powell took it well.

Of course Strategy 57 has an advantage by being able to essentially ‘run more horses’. The disadvantage is that it does not trigger often, which is why we need to ‘run more horses’ when it does. Still, it proves that the best way to take advantage of a low trade count, high win rate strategy is to push when it hits.

Strangely enough, I would have preferred to see how Strategy 57 responded when the market went against it, but hopefully we’ll have ~29 or so more opportunities to do just that in 2024.

Now let’s see what the entire portfolio did. Here are the results of the Forward test from January 1 of 2023 to January 16 of 2024: