Hi all,I wanted to get the forum's collective wisdom on my proposed methodology.I am writing a research proposal for testing the statistical significance of deterministic buy signals generated by a Technical Analysis (TA) indicator. The indicator in question doesn't have any parameters requiring a 'calibration' period, and possesses completely deterministic rules to generating buy/sell signals. (i.e. no subjective 'guessing' where a 'head' starts and a 'shoulder ends' etc). Similar in framework to Tom DeMark's indicators.Any feedback welcome.PROPOSED METHODOLOGYSTEP 1: Obtain market tick data for 100 daysThe indicator uses 1 hour bars of data but requires tick data for exact entry points. (See my earlier post regarding data source reliability of Bloomberg, Thomson Reuters.)STEP 2: Generate the TA Buy SignalsUsing the system rules I find "X" number of buy signals over the 100 day periodSTEP 3: Generate an identical number of buy signals randomly Using a random number generator I create "X" number of buy signals over the same period.STEP 4: Repeat STEP 3 until I have 1000 sets of data i.e. Generate 999 sets of randomly generated buy signals in addition to one set of TA buy signals. STEP 5: Calculate the P&L for all 1000 sets. STEP 6: Rank all the returns Note the percentile ranking given by the TA data.DATA ANALYSISThe random walk hypothesis implies that I expect to find no statistically significant difference between using a TA indicator or a random number generator to create buy signals.H(0) NULL HYPOTHESISWithin a 10% confidence level, the TA indicator system is the same as picking random BUY signals. Therefore if the TA returns are below the 90th percentile of ranked returns we cannot reject the null.H(1) ALTERNATIVE HYPOTHESISIf the technical indicator greater than the 90th percentile, we reject the null with a 10% confidence level. Note on Exit strategy and Stop Losses:The indicator gives a deterministic stoploss but not an exit strategy. Therefore I shall determine the exit strategy to be the same geometric return to the upside (i.e. a -5% downside stoploss would imply a +5% upside exit strategy).

the first problem I see is that you only have one path of your TA-strategy. maybe a null-set event ruined the performance of your strategy during that time.

Hmmm,The other way would be to bootstrap sections of the history to compare performance... alternatively I could run the same test over multiple markets, but then that assumes that all markets are identical in behaviour.Any suggestions?

I think your methodology makes sense, although I think statistical significance is less important than determining exit strategy. I understand that you arbitrarily choose +/-5% for exit and 1000 for the number of times you run the strategy, but you might try to play around with those values say use your indicator as stop loss / exit as well. You might also wanna consider Sharpe ratio in addition to PnL as it will help you once you wanna put the strategy out to the street. How do the results look so far with the hypothesis you suggested ?

edit