I would like to do something better to justify a trade/strategy. Currently I take price series, simulatea trading strategy, with some basic trading rules. I have statistics like Mean/Median/Stddev Trade countsetc for the strategy. I usualy use 1-1.5 years of data to backtest the strategy.I would really like to do better, Corelation tests, regression tests, if so between what, how do I approachthe problem.Some simple guidelines to building a stable framework really appreciated, since I am not a quant :-(I try to keep things simple (Keep it simple stupid).Regards

I definitely compare the correlations between various strategies (includes spreads) that I am running. It definitely reduces risk ... I try and restrain strategies so that their historic pair-correlations have been less than 0.30. It had definitely improved my trading.

I'm not sure what you mean.One danger with evaluating strategies from backtest is you may find strategies that work only in the past. If you try 1,000 strategies (or fewer but fit parameters for each using historical data), even if they perform only randomly, 50 of them will appear to work at the standard 5% level of statistical confidence.Even if you don't try so many strategies, there is a danger you will select a strategy that depends on the recent environment. For example, from 1991-93, you never lost money betting that interest rates would go down. As a result lots of people were highly levered up on that bet. They didn't think about it that way, and it was not generally known. But when interest rates went up, lots of disasters occured. In my experience, the same thing happens every time some market goes in one direction for three years.Therefore, it is worth checking the correlation of your strategy's return to simple long and short positions in all the relevant securities. Say you find you can make 50% per year, with a standard deviation of 10%, momentum trading stock A. That looks like a good strategy, five standard deviations above zero. But what if stock A had a 200% return per year with a standard deviation of 30%, and a correlation of 0.9 with your strategy? That means your strategy has a -10% expected return if stock A stays flat. Suddenly it doesn't seem so appealing. It's not a guaranteed money-maker, it's a levered bet on stock A. This is what killed a lot of people who day traded technology stocks from 1997-99.The trick is to look at a market-neutral version of your strategy. Assume you run it with fixed long and short positions such that the combined portfolio is uncorrelated with anything you can think of. Then if it has a historical excess return several standard deviations above zero, you have some confidence it will continue to do well in the future.

Thank you,I understand the first part of your reply,>Therefore, it is worth checking the correlation of your strategy's return to simple long and short positions in all the relevant securities. Say you find you can make 50% per year, with a standard deviation of 10%, momentum trading stock A. That looks like a good strategy, five standard deviations above zero. But what if stock A had a 200% return per year with a standard deviation of 30%, and a correlation of 0.9 with your strategy? That means your strategy has a -10% expected return if stock A stays flat. Suddenly it doesn't seem so appealing. It's not a guaranteed money-maker, it's a levered bet on stock A. This is what killed a lot of people who day traded technology stocks from 1997-99>WhatI should be doing is checking the correlation of my strategy returns to stock returns during that period.It would mean that my strategy's success would highly depend on this correlation being consistent during thenew period.Sorry but the second part eludes me.The trick is to look at a market-neutral version of your strategy. Assume you run it with fixed long and short positions such that the combined portfolio is uncorrelated with anything you can think of. Then if it has a historical excess return several standard deviations above zero, you have some confidence it will continue to do well in the future. Would appreciate help.Regards

Okay, here's an example. You have some trading system for a stock, and you compute the value of a hypothetical $1 million invested 18 months ago. You have an end-of-day value for each day in the period. That gives you a log return for each day. Compute the Beta of the strategy on the stock (and anything else that you think might be correlated).Say the Beta is -0.3, that is the strategy does better when the stock goes down. Then the portfolio of $1,000,000 in the strategy plus $300,000 of the stock will be uncorrelated with the stock price. The Sharpe ratio (or whatever performance measure you use) of this portfolio is more reliable than the Sharpe ratio of the strategy alone.The point is you want to subtract off from the strategy any component that you think is random. If you have no view about the direction of the stock, but the strategy does better when the stock goes one way or the other, you should remove that effect from the historical data. That tells you the pure effect of the strategy, free from the unintended side-effect correlations.

Thank you very much for your time. Your reply was very helpful.Regards

**One danger with evaluating strategies from backtest is you may find strategies that work only in the past. If you try 1,000 strategies (or fewer but fit parameters for each using historical data), even if they perform only randomly, 50 of them will appear to work at the standard 5% level of statistical confidence.**Aaron, wouldn't be the case that if you both use a larger sample (lets say more than 10 years ), and you take good care of testing your strategy out-of-sample, that should take care of testing the robustness of your strategy to different market regimes/conditions, and the robusteness of your estimated parameters?

A somewhat related question:Is there out there some trading platform where you can actually test your trading strategy in real-time with a virtual account. I know there are a couple of them for Spot FX (like OANDA, CMS visual trading- the only virtual thing there is the amount of money that is in your account, the rest - broker fees, interest rate charges,etc is exactly like in real market conditions), and I was wondering if there is something similar for futures and eventually options. And also, I was wondering if something like that will be helpful to test a strategy.Thanks

QuoteOriginally posted by: DogonMatrixAaron, wouldn't be the case that if you both use a larger sample (lets say more than 10 years ), and you take good care of testing your strategy out-of-sample, that should take care of testing the robustness of your strategy to different market regimes/conditions, and the robusteness of your estimated parameters?Out-of-sample testing is very important. It helps you against parameter-fitting bias. But it does not protect against selection bias if you test a lot of strategies. 10 years of data helps, not directly, but because you can set the individual significance level very low, say 0.000001. In that case even if you test 1,000 strategies, there is still some significance to your result. Robustness and testing in various regimes and conditions are good ideas as well, but they do not eliminate selection bias.The downside to all these safeguards is many good strategies will not get past them. What if the strategy doesn't work over 10 years, but does work over the last 3 months? What if it makes a lot of money, but not consistently enough for statistical proof. After all, if a strategy worked for 10 years and could be clearly shown to be profitable by conventional statistics, what is the chance that no one else has spotted it and exploited it away?Market strategies have to take some risk, they have to operate on the edge of statistical signficance. You backtest and analyze to find strategies that appear to work, but it still comes down to judgement and calculated risks.