Dear all, I have a question on the validity of backtesting process and results. Any input/suggestion/criticism is much appreciated!1. Suppose I have an indicator (let's use simple moving average as an example). I use 5, 10, 15, 20 days for SMA calculations, and use these SMAs to predict the return of following 5 trading days. 2. Say I have 30 yrs of historical daily prices. I use the first 20 yrs to examine, for each SMA (5, 10, 15, 20 days) the range of the SMAs which performed the best in predicting future 5-day returns (say 1.01 to 1.10 for 5-day SMA). Using the predefined range for each SMA in the historical analysis, I use the last 10 yrs of data to do an out-of-sample testing to examine the profitability of each SMA strategy. 3. Finally, assuming that 2 shows above-average profitability and 2 shows below average profitability in the out-of-sample results, I take the two good ones and use them as my trading strategy going forward. The theoretical reason for using the indicator is not important (SMA is just an example). Essentially this is a method of digging historical patterns and using out-of-sample study to validate and identify the optimal strategy. I would appreciate any comment on how to improve or correct the backtesting / strategy evaluation process. Thanks so much!!David

out-of-sample forecasts are a good test for the type of model u r talking about. the real trick in success going forward using this methodology is understanding the market's dynamics i.e. how far back in time should your time-series go, does your model hold true during different time periods, what variables were more important and why during each of your samples. you'll notice that during certain periods some variables track better than others, you need to know what triggers the switch. this will allow you to forecast more accurately going forward.

QuoteOriginally posted by: kickassDear all, I have a question on the validity of backtesting process and results. Any input/suggestion/criticism is much appreciated!1. Suppose I have an indicator (let's use simple moving average as an example). I use 5, 10, 15, 20 days for SMA calculations, and use these SMAs to predict the return of following 5 trading days. 2. Say I have 30 yrs of historical daily prices. I use the first 20 yrs to examine, for each SMA (5, 10, 15, 20 days) the range of the SMAs which performed the best in predicting future 5-day returns (say 1.01 to 1.10 for 5-day SMA). Using the predefined range for each SMA in the historical analysis, I use the last 10 yrs of data to do an out-of-sample testing to examine the profitability of each SMA strategy. 3. Finally, assuming that 2 shows above-average profitability and 2 shows below average profitability in the out-of-sample results, I take the two good ones and use them as my trading strategy going forward. The theoretical reason for using the indicator is not important (SMA is just an example). Essentially this is a method of digging historical patterns and using out-of-sample study to validate and identify the optimal strategy. I would appreciate any comment on how to improve or correct the backtesting / strategy evaluation process. Thanks so much!!DavidIMHO, If you do it this way, you will rarely ever find a signal that behaved consistently. In fact suppose you have a 30 yr history and u optimize a signal over the first 20 years and then even if you break up the 20 years into segments of 5 years each, you might just find that the top performing signal probably didn't work for one or more of the 5 year period ... or atleast often there will be a 2-3 years period when any of the top performing signals didn't work ... In a real trading environment you can get fired if you don't produce profitable results for 2-3 years at a stretch ... combining multiple technical signals can help to lesson the effect - but my experience with them have been that the technical signals are often correlated to the extent that when one fails, the others fail too etc. A better way is to come up with a model that can identify periods when a particular signals is unlikely to perform well. For instance, a model could be to identify during which periods say a shorter 10 day SMA (or whatever) will work and in which period a longer say 30 day SMA will work and automatically use the signal based on the regimes etc and so on ...

QuoteOriginally posted by: kickass3. Finally, assuming that 2 shows above-average profitability and 2 shows below average profitability in the out-of-sample results, I take the two good ones and use them as my trading strategy going forward.What percentage showed profitability in the in-sample results? In other words, suppose you tried eight different moving averages. Probably four would make money. If you then tried any four of them on another price series, probably two of those would make money.Of course adding new tests will narrow the set of strategies which pass all of them. But does backtesting show any tendency to select strategies which are more likely to pass each new test?

Thank you all for your help!David