Oct 9, 2022·edited Oct 9, 2022Liked by Celan Bryant (CB)
I'm sorry if I missed something obvious, but I cannot see anywhere I've looked -- including the source code of your strat #1 -- are your strategies applied to a single security, or a pool? Could you please either point me to the description of how the strategies are applied to the asset(s)?
Secondly, while back-testing is a reasonable tool, what I would LOVE to see is how the strategies worked when applied to a validation or -- better -- a test period. While I am relatively new to the data science world, I have come to understand that performance degrades from training to validation to test -- depending on the strength of an algorithm.
Lastly, I think it would be appropriate to report on percentage of gains vs absolute dollar amounts. I don't know what the basis is; so how can I assess the net gain performance?
The first link is the most recent performance chart that gets updated every two months. The second link is to our most recent update.
We are concentrated on futures contracts now, but hope to branch out to other assets in 6 months or so.
Most strategies are based on running a single asset, however, we've just started testing portfolios as well. You can read about how that's going in the most recent Mudder Report:
You are right about performance degradation. It is something we talk about a lot in updates. Backtest accuracy has plagued us religiously, but we've figured out a way to carve out about 85% of the associated risk in development. Again, all of this is documented in the updates.
The metrics we use to monitor performance have been a work-in-process and are very different from when we first started. You can scroll to the bottom of either link above -- (1) or (2) -- for metric definitions. For example, we use profit factor now (gross profit/gross loss) as our primary attribute. I also give you the gross profit and gross loss, along with the drawdown, so you can get a better feel for how the strategy works. I also give you the percentage of profitable trades, max/min daily low/high, max/min cumulative low/high and various per trade stats. Take a look at the most recent performance chart in (1) above.
Hope that helps. Let me know if you still have any questions.
An article referenced under the question, "What is overfitting and what are you doing about it?," is locked behind a paywall. Does this article, "How Can You Tell If An Automated Trading Strategy Will Perform Well Over Time?," explain how Ninjatrader's Walk Forward optimization method is leveraged in ATS development?
Assuming some type of walk forward analysis is utilized, why compare the backtest results of strategies instead of the combined OOS walk forward series for each strategy?
Thank you for the question/suggestion. We've had a lot of difficulty with backtest accuracy so at the beginning of September, I started a real-time test on a virtual server to compare to backtest results. Those results are shared every week in the Mudder Report.
Also, as a quick note, we don't optimize on indicator parameters -- it leads to overfitting. However, just recently, I started using NT8s walk-forward optimization analysis to test optimization strategies based on time series. I got some great results, but then we still have the issue of accuracy with real-time data. It always seems to come back to that.
In your research, have you found the walk-forward optimization to be more accurate than the backtest? How are you using it? Have you found any success with any of the other tools available in the Strategy Analyzer?
Thanks for your response. I appreciate the questions.
I understand you are looking for a measure of return based on percentage of investment. We use return on max drawdown as a proxy. It is the 5th column in the performance chart. The date of the performance chart is one year (the exact dates are highlighted in yellow). It is assumed that all accounts start at $0, though in reality you would need a minimum account balance to get started. This is why we suggest a starting account value of max drawdown plus a small buffer for margin. See below for metric definitions.
I also noticed that the total/average column that is usually at the top of each portfolio update was omitted in September. I think that might be part of the issue here too as you reference $2.7MM in net profit. So, I've made that correction. If you go to the update now, you will find the total/average rows at the top of the portfolio. You will also find a total portfolio return on max drawdown, which is 633%. This is based on a total portfolio max drawdown of $418K. Also, keep in mind that these are all backtested numbers. We're currently working on duplicating these results using real-time data.
Thanks again for the feedback. Let me know if you're looking for something more.
Celan
Drawdown - This refers to the maximum drawdown statistic, which provides you with information regarding the biggest decrease (drawdown) in account size experienced by the strategy. Drawdown is often used as an indicator of risk.
Drawdown = single largest Drawdown
As an example, your account rises from $25,000 to $50,000. It then subsequently drops to $40,000 but rises again to $60,000. The drawdown in this case would be $10,000 or -20%. Take note that drawdown does not necessarily have to correspond with a loss in your original account principal.
Return on Max Drawdown - We’ve added a dollar value for max drawdown along with a measure of return (return on max drawdown), which is calculated by dividing net profit by the max drawdown. In this way, max drawdown is considered the max capital investment. You can use the dollar value of max drawdown as a proxy for how much capital you need to trade the strategy. And, the higher the return on max drawdown, the better the strategy is in terms of risk/reward.
Hi Damian, I thought I answered this already. My apologies for the delay.
The best VPS by definition is the one that never goes offline. So downtime is the most important selling point and it's what most vendors reference in terms of capabilities. That said, in my experience, the best provider is one that responds back to you within 5 minutes. That's my personal test. Hope that helps.
Thanks for the response Celan. I read the Strategy Descriptions page, the September update, and the Mudder Report. I also did a bit of research on performance metrics. I see the merit (and simplicity) of the Profit Factor, but I still think performance metrics should: (1) be reported as a percent of the investment, (2) be reported on an annualized basis, and (3) be compared to the underlying asset.
On point #1: I am still unclear what the basis (investment) was when reporting a $2.7MM net profit in the September Update. How can I put that into context with any other investments? In order to make investment decision, one must be able to compare performance across different vehicles. Right?
Point #2: without understanding the investment period, we cannot compare across different vehicles.
Point #3: Compared to buy & hold, strategies are complicated and can be riskier. As a result, the best comparison of both performance and risk is against the buy & hold of the underlying asset. No?
I hope you take these perspectives into considering as you continue to develop your system.
I'm sorry if I missed something obvious, but I cannot see anywhere I've looked -- including the source code of your strat #1 -- are your strategies applied to a single security, or a pool? Could you please either point me to the description of how the strategies are applied to the asset(s)?
Secondly, while back-testing is a reasonable tool, what I would LOVE to see is how the strategies worked when applied to a validation or -- better -- a test period. While I am relatively new to the data science world, I have come to understand that performance degrades from training to validation to test -- depending on the strength of an algorithm.
Lastly, I think it would be appropriate to report on percentage of gains vs absolute dollar amounts. I don't know what the basis is; so how can I assess the net gain performance?
Thanks in advance.
Hi Bob,
There's a lot of info I know, so I'm going to point you to two places:
1) https://automatedtradingstrategies.substack.com/p/strategy-descriptions
2) https://automatedtradingstrategies.substack.com/p/automated-trading-strategies-september-5fb
The first link is the most recent performance chart that gets updated every two months. The second link is to our most recent update.
We are concentrated on futures contracts now, but hope to branch out to other assets in 6 months or so.
Most strategies are based on running a single asset, however, we've just started testing portfolios as well. You can read about how that's going in the most recent Mudder Report:
https://automatedtradingstrategies.substack.com/p/the-mudder-report-20-100222-100722
You are right about performance degradation. It is something we talk about a lot in updates. Backtest accuracy has plagued us religiously, but we've figured out a way to carve out about 85% of the associated risk in development. Again, all of this is documented in the updates.
The metrics we use to monitor performance have been a work-in-process and are very different from when we first started. You can scroll to the bottom of either link above -- (1) or (2) -- for metric definitions. For example, we use profit factor now (gross profit/gross loss) as our primary attribute. I also give you the gross profit and gross loss, along with the drawdown, so you can get a better feel for how the strategy works. I also give you the percentage of profitable trades, max/min daily low/high, max/min cumulative low/high and various per trade stats. Take a look at the most recent performance chart in (1) above.
Hope that helps. Let me know if you still have any questions.
Celan
An article referenced under the question, "What is overfitting and what are you doing about it?," is locked behind a paywall. Does this article, "How Can You Tell If An Automated Trading Strategy Will Perform Well Over Time?," explain how Ninjatrader's Walk Forward optimization method is leveraged in ATS development?
Assuming some type of walk forward analysis is utilized, why compare the backtest results of strategies instead of the combined OOS walk forward series for each strategy?
Hi Oliver,
Thank you for the question/suggestion. We've had a lot of difficulty with backtest accuracy so at the beginning of September, I started a real-time test on a virtual server to compare to backtest results. Those results are shared every week in the Mudder Report.
Also, as a quick note, we don't optimize on indicator parameters -- it leads to overfitting. However, just recently, I started using NT8s walk-forward optimization analysis to test optimization strategies based on time series. I got some great results, but then we still have the issue of accuracy with real-time data. It always seems to come back to that.
In your research, have you found the walk-forward optimization to be more accurate than the backtest? How are you using it? Have you found any success with any of the other tools available in the Strategy Analyzer?
Thanks,
Celan
Hello again Bob,
Thanks for your response. I appreciate the questions.
I understand you are looking for a measure of return based on percentage of investment. We use return on max drawdown as a proxy. It is the 5th column in the performance chart. The date of the performance chart is one year (the exact dates are highlighted in yellow). It is assumed that all accounts start at $0, though in reality you would need a minimum account balance to get started. This is why we suggest a starting account value of max drawdown plus a small buffer for margin. See below for metric definitions.
I also noticed that the total/average column that is usually at the top of each portfolio update was omitted in September. I think that might be part of the issue here too as you reference $2.7MM in net profit. So, I've made that correction. If you go to the update now, you will find the total/average rows at the top of the portfolio. You will also find a total portfolio return on max drawdown, which is 633%. This is based on a total portfolio max drawdown of $418K. Also, keep in mind that these are all backtested numbers. We're currently working on duplicating these results using real-time data.
Thanks again for the feedback. Let me know if you're looking for something more.
Celan
Drawdown - This refers to the maximum drawdown statistic, which provides you with information regarding the biggest decrease (drawdown) in account size experienced by the strategy. Drawdown is often used as an indicator of risk.
Drawdown = single largest Drawdown
As an example, your account rises from $25,000 to $50,000. It then subsequently drops to $40,000 but rises again to $60,000. The drawdown in this case would be $10,000 or -20%. Take note that drawdown does not necessarily have to correspond with a loss in your original account principal.
Return on Max Drawdown - We’ve added a dollar value for max drawdown along with a measure of return (return on max drawdown), which is calculated by dividing net profit by the max drawdown. In this way, max drawdown is considered the max capital investment. You can use the dollar value of max drawdown as a proxy for how much capital you need to trade the strategy. And, the higher the return on max drawdown, the better the strategy is in terms of risk/reward.
What’s your favourite VPS service provider? What to look for when choosing one?
Hi Damian, I thought I answered this already. My apologies for the delay.
The best VPS by definition is the one that never goes offline. So downtime is the most important selling point and it's what most vendors reference in terms of capabilities. That said, in my experience, the best provider is one that responds back to you within 5 minutes. That's my personal test. Hope that helps.
Trade well,
Celan
Thanks for the response Celan. I read the Strategy Descriptions page, the September update, and the Mudder Report. I also did a bit of research on performance metrics. I see the merit (and simplicity) of the Profit Factor, but I still think performance metrics should: (1) be reported as a percent of the investment, (2) be reported on an annualized basis, and (3) be compared to the underlying asset.
On point #1: I am still unclear what the basis (investment) was when reporting a $2.7MM net profit in the September Update. How can I put that into context with any other investments? In order to make investment decision, one must be able to compare performance across different vehicles. Right?
Point #2: without understanding the investment period, we cannot compare across different vehicles.
Point #3: Compared to buy & hold, strategies are complicated and can be riskier. As a result, the best comparison of both performance and risk is against the buy & hold of the underlying asset. No?
I hope you take these perspectives into considering as you continue to develop your system.