Oct 9, 2022·edited Oct 9, 2022Liked by Celan Bryant (CB)
I'm sorry if I missed something obvious, but I cannot see anywhere I've looked -- including the source code of your strat #1 -- are your strategies applied to a single security, or a pool? Could you please either point me to the description of how the strategies are applied to the asset(s)?
Secondly, while back-testing is a reasonable tool, what I would LOVE to see is how the strategies worked when applied to a validation or -- better -- a test period. While I am relatively new to the data science world, I have come to understand that performance degrades from training to validation to test -- depending on the strength of an algorithm.
Lastly, I think it would be appropriate to report on percentage of gains vs absolute dollar amounts. I don't know what the basis is; so how can I assess the net gain performance?
An article referenced under the question, "What is overfitting and what are you doing about it?," is locked behind a paywall. Does this article, "How Can You Tell If An Automated Trading Strategy Will Perform Well Over Time?," explain how Ninjatrader's Walk Forward optimization method is leveraged in ATS development?
Assuming some type of walk forward analysis is utilized, why compare the backtest results of strategies instead of the combined OOS walk forward series for each strategy?
Thanks for your response. I appreciate the questions.
I understand you are looking for a measure of return based on percentage of investment. We use return on max drawdown as a proxy. It is the 5th column in the performance chart. The date of the performance chart is one year (the exact dates are highlighted in yellow). It is assumed that all accounts start at $0, though in reality you would need a minimum account balance to get started. This is why we suggest a starting account value of max drawdown plus a small buffer for margin. See below for metric definitions.
I also noticed that the total/average column that is usually at the top of each portfolio update was omitted in September. I think that might be part of the issue here too as you reference $2.7MM in net profit. So, I've made that correction. If you go to the update now, you will find the total/average rows at the top of the portfolio. You will also find a total portfolio return on max drawdown, which is 633%. This is based on a total portfolio max drawdown of $418K. Also, keep in mind that these are all backtested numbers. We're currently working on duplicating these results using real-time data.
Thanks again for the feedback. Let me know if you're looking for something more.
Celan
Drawdown - This refers to the maximum drawdown statistic, which provides you with information regarding the biggest decrease (drawdown) in account size experienced by the strategy. Drawdown is often used as an indicator of risk.
Drawdown = single largest Drawdown
As an example, your account rises from $25,000 to $50,000. It then subsequently drops to $40,000 but rises again to $60,000. The drawdown in this case would be $10,000 or -20%. Take note that drawdown does not necessarily have to correspond with a loss in your original account principal.
Return on Max Drawdown - We’ve added a dollar value for max drawdown along with a measure of return (return on max drawdown), which is calculated by dividing net profit by the max drawdown. In this way, max drawdown is considered the max capital investment. You can use the dollar value of max drawdown as a proxy for how much capital you need to trade the strategy. And, the higher the return on max drawdown, the better the strategy is in terms of risk/reward.
Thanks for the response Celan. I read the Strategy Descriptions page, the September update, and the Mudder Report. I also did a bit of research on performance metrics. I see the merit (and simplicity) of the Profit Factor, but I still think performance metrics should: (1) be reported as a percent of the investment, (2) be reported on an annualized basis, and (3) be compared to the underlying asset.
On point #1: I am still unclear what the basis (investment) was when reporting a $2.7MM net profit in the September Update. How can I put that into context with any other investments? In order to make investment decision, one must be able to compare performance across different vehicles. Right?
Point #2: without understanding the investment period, we cannot compare across different vehicles.
Point #3: Compared to buy & hold, strategies are complicated and can be riskier. As a result, the best comparison of both performance and risk is against the buy & hold of the underlying asset. No?
I hope you take these perspectives into considering as you continue to develop your system.
I'm sorry if I missed something obvious, but I cannot see anywhere I've looked -- including the source code of your strat #1 -- are your strategies applied to a single security, or a pool? Could you please either point me to the description of how the strategies are applied to the asset(s)?
Secondly, while back-testing is a reasonable tool, what I would LOVE to see is how the strategies worked when applied to a validation or -- better -- a test period. While I am relatively new to the data science world, I have come to understand that performance degrades from training to validation to test -- depending on the strength of an algorithm.
Lastly, I think it would be appropriate to report on percentage of gains vs absolute dollar amounts. I don't know what the basis is; so how can I assess the net gain performance?
Thanks in advance.
An article referenced under the question, "What is overfitting and what are you doing about it?," is locked behind a paywall. Does this article, "How Can You Tell If An Automated Trading Strategy Will Perform Well Over Time?," explain how Ninjatrader's Walk Forward optimization method is leveraged in ATS development?
Assuming some type of walk forward analysis is utilized, why compare the backtest results of strategies instead of the combined OOS walk forward series for each strategy?
Hello again Bob,
Thanks for your response. I appreciate the questions.
I understand you are looking for a measure of return based on percentage of investment. We use return on max drawdown as a proxy. It is the 5th column in the performance chart. The date of the performance chart is one year (the exact dates are highlighted in yellow). It is assumed that all accounts start at $0, though in reality you would need a minimum account balance to get started. This is why we suggest a starting account value of max drawdown plus a small buffer for margin. See below for metric definitions.
I also noticed that the total/average column that is usually at the top of each portfolio update was omitted in September. I think that might be part of the issue here too as you reference $2.7MM in net profit. So, I've made that correction. If you go to the update now, you will find the total/average rows at the top of the portfolio. You will also find a total portfolio return on max drawdown, which is 633%. This is based on a total portfolio max drawdown of $418K. Also, keep in mind that these are all backtested numbers. We're currently working on duplicating these results using real-time data.
Thanks again for the feedback. Let me know if you're looking for something more.
Celan
Drawdown - This refers to the maximum drawdown statistic, which provides you with information regarding the biggest decrease (drawdown) in account size experienced by the strategy. Drawdown is often used as an indicator of risk.
Drawdown = single largest Drawdown
As an example, your account rises from $25,000 to $50,000. It then subsequently drops to $40,000 but rises again to $60,000. The drawdown in this case would be $10,000 or -20%. Take note that drawdown does not necessarily have to correspond with a loss in your original account principal.
Return on Max Drawdown - We’ve added a dollar value for max drawdown along with a measure of return (return on max drawdown), which is calculated by dividing net profit by the max drawdown. In this way, max drawdown is considered the max capital investment. You can use the dollar value of max drawdown as a proxy for how much capital you need to trade the strategy. And, the higher the return on max drawdown, the better the strategy is in terms of risk/reward.
What’s your favourite VPS service provider? What to look for when choosing one?
Thanks for the response Celan. I read the Strategy Descriptions page, the September update, and the Mudder Report. I also did a bit of research on performance metrics. I see the merit (and simplicity) of the Profit Factor, but I still think performance metrics should: (1) be reported as a percent of the investment, (2) be reported on an annualized basis, and (3) be compared to the underlying asset.
On point #1: I am still unclear what the basis (investment) was when reporting a $2.7MM net profit in the September Update. How can I put that into context with any other investments? In order to make investment decision, one must be able to compare performance across different vehicles. Right?
Point #2: without understanding the investment period, we cannot compare across different vehicles.
Point #3: Compared to buy & hold, strategies are complicated and can be riskier. As a result, the best comparison of both performance and risk is against the buy & hold of the underlying asset. No?
I hope you take these perspectives into considering as you continue to develop your system.