How To Set Up Your Own Forward Test
Forward Test Challenge 2.0
Important: There is no guarantee that our strategies will have the same performance in the future. We use backtests to compare historical strategy performance. Backtests are based on historical data, not real-time data so the results we share are hypothetical, not real. There are no guarantees that this performance will continue in the future. Trading futures is extremely risky. If you trade futures live, be prepared to lose your entire account. We recommend using our strategies in simulated trading until you/we find the holy grail of trade strategy. To see the most recent backtest of all ATS strategies, along with strategy description links, click here.
[Update 12/30/2023] If you are interested in an alternative way to set up a forward test without the use of a virtual server, you can use a cell phone data failover like the Netgear 4G LTE (use as a failover solution for always-on WiFi Certified). You can use pay for usage data cards to avoid having to pay for a fixed data plan. If the internet goes down, usage goes up, but no usage otherwise. (Thanks again Stuart)
If anyone has other ways to save money on live data and/or virtual servers, please let me know.
When we first started, I published 10 strategies that barely had a profit factor greater than 1.05. Today, we have 75+ strategies, and the average profit factor is well over 2.00.
For an overview of our progress, you can review the updates below.
I’ll provide the next quarterly update in January.
Throughout the last 3 years, the most common question I get after someone subscribes is still the same: what’s the best strategy to trade?
My answer to that question has evolved. I used to give people a strategy. Then when it started to perform poorly, they would come back and ask me for another strategy. Then another.
Why is this?
It is the nature of automated strategies to have hot and cold runs. One thing we’re trying to do now is hack the way in which strategies seem to perform well at different frequencies. Since we’re primarily using minutes, the frequency refers to the amount of time it takes for one bar to form. Clearly, it takes more time for a 60 minute bar to form than a 1 minute bar. What I’ve noticed is that a strategy will perform well at the 30 minute data series one week and the 45 minute data series the next. Larry, writer of Hunt, Gather, Trade refers to it as “frequency hopping”. After much research, it seems that those traders that use automated trading strategies on a regular basis are more like DJs. They find a robust strategy and run it on the best frequency.
Here’s an example of a chart that tracks the profit factor of a strategy from 1 to 60 minutes. There’s a clear pattern here. It seems like every 10 minutes there’s a peak (thank you for that amazing observation Kevin).
We’re just starting to study this, so I’ll keep you posted on any relevant findings. Needless to say, the best way to study this phenomena is with a forward test. In fact, a forward test is the best way to study all things ‘automated trading’. I don’t even think you can really say you’re taking the hunt seriously until you start your own forward test.
So that’s a roundabout way of explaining why my answer to the question “what’s the best strategy to run?” has evolved to: start a forward test with 15-20 strategies using live data on simulated accounts. Ninjatrader allows you to create an unlimited number of sim accounts. Over time, you will be able to answer your own question over and over again. This is the best way to leapfrog all the work ATS has done over the past 3 years. It took us over a year to start a forward test, and six additional months to really take the effort seriously.
There are currently over 75 strategies here that you can download and start running on a simulated account today. You can also develop your own strategies or variations on ours. Some of the best strategies I’ve received were variations on an ATS strategy, which brings me to the following CHALLENGE:
The Challenge: A few days ago I received an email from a new subscriber telling me that he planned on taking me up on my forward test challenge.
What’s the forward test challenge?
If you can find a strategy with 10 days of consecutive positive gains in a forward test, I will comp your subscription.
Another common question I receive is: Why is a simulated live forward test better than a backtest?
No historical data set is ever going to be better than live data. When you use historical data you lose data; literally. The further back you go, the more data you lose. It takes a great deal of resources to simulate reality, even one as simple as the NQ market, but that’s where we are in the hunt. Historical data is also prone to gaps. There are arguments to be had for the use of raw or refined data; the former comes straight from the exchange, while the latter is tampered with, but closer to reality. I lean toward the latter argument, but this is a moot point if you’re using live data.
So the best approach is to use the backtest to point you in the right direction and the forward test to see if that direction is right. The glorious thing is that you don’t have to fund an account to do so—you can use a simulated account on live data until you’ve found that perfect strategy (or strategies). What I’m about to share with you is how I created my own forward test. You don’t have to do yours this way—it’s only a suggestion.
How To Create Your Own Forward Test
Before I get into how I created my forward test, I want to provide a quick overview of my experience with creating and running a forward test over the last year.
I started the forward test in January of 2023 and it continues today. Clearly, it has become my favorite tool for learning more about how automated strategies operate. Just like in permaculture, you have to observe something before you can understand it.
I’ve observed many things including:
which strategies are correlated with one another
which strategies are on a hot run and which are running cold
which instruments are in hot markets and which are in cold or stagnant markets
which strategies are more robust that others
which strategies trade well in certain markets
You may say that you can test any of these by looking at the backtest, which is true. The issue is that the data you’re using in the backtest is of a lower quality than live data; data is lost in translation, and the result is inaccurate backtests and lost account funds. If you ask me, this is why so many people lose money—they go from backtest to real money without a forward test.
Not all backtests are inaccurate. In fact, my goal over the last year and half has been to lean into those elements in strategies that are more efficient and/or easier for the simulation to handle. The hope being that it will produce more accurate backtests. The use of Ninjatrader’s AI Generate has also helped with this effort.
Now let’s get into the steps I used to set up my own forward test using Ninjatrader.
I was not involved in all aspects of the set up, but these are the basic steps:
Find a virtual server located as close to the exchange where the asset you’re trading is located. I know some of you don’t feel this step is necessary. In fact, I just updated this post (see update above), with an alternative method. What I have found is that it is very difficult to create an environment that you can count on to run 24/7. I can’t tell you how many times I’ve created a portfolio of strategies only to have some type of system issue that requires me to rebuild the portfolio. It is a truly humbling experience. The virtual server mitigates this risk.
You want to download and run NT8 from within this virtual server. The best virtual server is one that is never down (read: zero downtime). Even more important is that they respond to you within 5 minutes if you do have an issue.
If you trade on your own account, you might need to purchase a second account from your brokerage so you can run both accounts simultaneously without one instance bumping the other out. You may also need to buy another data package. I use NT brokerage. If you contact NT brokerage, they will walk you through the process.
You also need to be mindful of RAM. We recently had to upgrade to 16gb. That said, we’re running over 130 strategies, so this is something that depends on resource use. Your provider should have a way for you to monitor your resource usage in real-time. Look at peak hours to know your minimum requirements.
You will need to create a backup file of your current settings and import it onto the new server. Then, copy and paste the backup file onto the new server to import. Here's how to copy your existing NinjaTrader workspace, charts, indicators, etc. NinjaTrader Documentation for exporting a backup: https://ninjatrader.com/support/helpGuides/nt8/creating_a_backup_archive.htm
If the strategies aren’t included in the backup file, you will need to import our (or your own) strategies into NT8 on the virtual server. Ninjatrader download instructions are listed below for downloading our strategies from the Strategy Description:
Click on the link within the strategy description to download the file from G drive.
Download the Strategy to your desktop, keep them in the compressed .zip file.
From the NinjaTrader Control Center window, select the menu Tools > Import > NinjaScript Add-On
Select the downloaded file from your desktop
After you find a virtual server, download NT8, and import strategies, but before you connect to your account to set the strategy up, you will need to create simulated accounts to run the strategy on. This can be done when you are connected, but you will need to disconnect and then reconnect before the accounts can be used.
All accounts come with one simulated account preset called SIM, but you can create as many as you want. If you click the “create a simulated account” link above, it will take you to a quick tutorial on creating new simulated accounts.
I like for accounts to have the same name as the strategy it will be run on, i.e. SimAccount2 for Strategy 2 (and all its variations). I also like to create accounts for portfolios, ie. SimPortfolio54b, and start all accounts out at $50K for the sake of comparison. You can also set starting account size to the cumulative max drawdown of the strategy if you’re looking for greater accuracy.
Now you can connect and start running the strategy. If you’ve never run a strategy before, NT8 provides a great overview here. Even if you’ve run a strategy before, this is a great overview of what options you have available to you.
There are two main ways to run a strategy: from a chart and from the Strategies tab. We use both. Personally, I find the chart to be easier when setting up more than 3 strategies, but the Strategy tab is better for adding one or two strategies at a time. It’s good to practice on both to see which one you prefer. Setting up strategies on a chart does have an additional advantage. It allows your strategies to auto-update to the current front-month. If you start a strategy from the Strategies tab, it will not update automatically.
The best time to start getting set up is after the market closes on Friday evening. You need to make sure everything is ready to go by Sunday at 6pm (EST).
Monitor your strategies closely. You want to begin with the end in mind (as the saying goes). That means you want to start thinking like a portfolio manager. Everything about the forward test should be real/live except for the account value. Compare the performance against the backtest on a daily basis at first and track their performance over time like we’re doing. We’ve set up an automated email for a trade performance report. That email goes to an email address dedicated to trade performance reports. I requested this to 1) verify that the daily trade performance report is working properly 2) maintain a daily record of trade performance in case the database becomes corrupted. This actually happened, which is why I added this step.
Once you get set up, you want to develop a strategy to run the portfolio. Running a simulated account will help to develop your own management style. These are some things you need to think about:
What happens if you get kicked out of a strategy? This can happen for many reasons, but it usually happens when you have two conflicting strategies on the same contract. This is why we use the same strategy on multiple instruments, but not the same instrument on multiple strategies. If you need help with this, shoot me an email.
What happens if the strategy falls out of sync? There are many reasons that a strategy may be out of sync. This is usually due to a disconnect between the account position and the strategy position. You can learn more about what it is and how to fix it here. If you need help with this, shoot me an email.
What’s your rollover strategy? Rollover is tricky. In my regular trading, I roll with contract volume. That is, the contract with the highest volume is the one I’m trading. Ninjatrader has an automated rollover system, but it’s static. You’ll want to close out of the application once a week for rollovers to update automatically. We’ve found that many more errors occur during rollover. Until the rollover system is more dynamic, we’re going to turn all strategies off during the last two weeks of major contract expiration dates (four times a year — triple witching) to “manage” the portfolio. In other words, no strategies are running for two weeks running up to triple witching. This might change in the future.
What happens if the virtual server shuts down?
Contact the provider immediately. If they don’t respond in 5 minutes, ask for a refund and find another provider. Don’t wait until this happens to test the 5 minute response rule.
What happens if your local computer goes offline? Nothing. This is why you have the strategies set up on a dedicated virtual server. If your local computer goes offline, it will not impact the performance of your strategies, but you should have some way to sign on to NT8 through another device, ie phone, tablet.
When you start the strategy, will you enter the trade immediately or will you wait until the strategy is flat? This is up to you. It is among the many options you have when you first set the strategy up. All of our strategies are set to wait until the strategy is flat, but I haven’t conducted any tests to see if one is better than the other from a performance perspective. Some of you have also said that calculating on each tick is better than calculating on bar close. I don’t know if that’s the case, but it certainly requires more resources to run. Here’s a screenshot of the default view for most of our strategies:
Do you want a strategy that closes at the end of the session or end of week? Some strategies have a better performance when run continuously through the week (read: you don’t automatically close all positions at session close). This also depends on the market and/or exchange. The futures market closes every day for one hour. If you want to hold your position through that close, you will need to add the cost of overnight margin to your starting account balance (i.e., cumulative max drawdown + initial margin). You can learn more about the impact of futures margin on automated trading here.
I’m sure I left something out, but those are the high level basics.
The more you live-test, the more you’ll develop your own sense for what works. If I missed something or if you have any other questions, please let me know. I’d prefer it if you would add your question/comment below so we can create a working document, but if you’re uncomfortable with that I’m okay with a direct email as well.
I also want to say congrats to Xa on his new job. That makes two assistants that have been picked off in the last year—I’m sad to see you go (and a bit upset because I don’t want to train anyone else), but honored to have worked with you as long as I did.
Now, I’m going to say something for subscribers only. This is how I get double value out of my forward tests: