I would like to kick off by saying that I dont know much about QF compared to most (perhaps all) of the people on here, so be prepared to forgive some ignorance.On another forum someone has said that, if I backtest a mechanical trading system, and over the series of trades the results show that the probabiliy of a win is 50% over the series. The probability of the any individual trade (or future trade) is not neccessarily 50% because the probability varies over time. (I think that this is Simpson's Paradox??")The argument is that unlike flipping a coin or the Lottery, this is a non-stationary process.Can someone give me some validity on this please?Thanks

If the question is, that a backtests yields probability of win = 0.5 then why doesn't the next draw have probability 0.5? 1. Well, forget backtesting for a second. The answer is maybe it does or maybe it doesn't. It depends on your model but doesn't necessarilly have to do anything with time. probabilities are long term creatures (they are limits).2. Now bring in backtesting. Perhaps what that other person meant was that by the time you do the next trade your probability distribution will have changed right under you. That's realistic. Sounds like the real world to me. But this neither good or bad since the odds can also shift in your favor! Problem is, how do you know that? Perhaps that can be backtested too

My practical experiance teels me that probabilities change. I backtested from 1988 - 1996 and the probability was 50%, then 1996 - 2004 was 30%. This proves to me that the next outcome of the next trade is random to a degree.Thanks for your reply and confirmation! Jamie

QuoteOriginally posted by: DaGuyPerhaps what that other person meant was that [...] your probability distribution will have changed right under you.the technical name of this being hidden markov models (HMM). Quite cool stuff, they are applied to speech recognition software - I had a go at this just for funI was wondering, are HMM's used in quantitative finance?Alex

There are several things you could be talking about here.The simplest is that the probability varies over time, with an average rate of 50%. Suppose you were betting on sports teams, and wanted to test the trading rule of always betting the team whose name came first in alphabetical order. Over any long series of bets, you would expect to win half. But in any individual contest, you could have a higher or lower probability. As DaGuy pointed out, this can work in your favor as often as against you.Another topic is fitting. If you estimate the parameters of your trading system based on historical information, and backtest it on the same information, your performance measurement is biased high. You expect actual performance to be worse than the backtest. This always works against you.Simpson's Paradox is a little different. Suppose I am comparing two trading systems over the last 1,000 trading days. Each system gives me a long/short signal each day. System A has an average return of 1.5% on long days and 0.5% on short days. System B has an average return of 1.4% on long days and 0.4% on short days. Which trading system is better?A naive person says "A" because it does better when it gives a long signal and better when it gives a short signal. But suppose system A gave 100 long signals and 900 short signals over the 1,000 tradings days; while system B gave 900 long and 100 short. A's average return is 0.6%, B's average return is 1.3%.

QuoteOriginally posted by: AaronThere are several things you could be talking about here.... If you estimate the parameters of your trading system based on historical information, and backtest it on the same information, your performance measurement is biased high. That is indeed very interesting. Could you please expand on this further? I understand what you are saying but I'm raking my brain trying figure out why that is the case, from a theoretical perspective. From a practical perspective, all I can think of is survivorship bias. If that is the case, what if we scrub the data as to minimize or eliminate it?Thanks in advance!

Let's suppose you are trying to beat a roulette wheel. A perfect wheel has 1/38 chance for each number. But say the wheels in one casino are slightly off balance. 28 of the numbers are selected with probability 0.025 each and 10 of the numbers are selected with probability 0.030 each.You record 200 spins of such a wheel, in which case you expect each of the 28 low-probability numbers to come up 5 times and each of the high probability numbers to come up 6 times. So you decide to bet in the future on every number that came up 6 or more times in the sample.Your backtest will show, on average, a 33% return on this strategy. But if you actually try it in the future, you will get, on average, a -4% return.Why? The trouble is that you are using the same data to decide which numbers to bet on and to evaluate the strategy. 38% of the low probability numbers will have 6 or more hits in 200, 44% of the high-probability numbers will have 5 or fewer hits. So in the future you will be betting on twice as many low-probability numbers (38% x 28) than high-probability ones (56% x 10).However, even if you correctly identified the high probability numbers and bet on only those 10, you would only get an 8% average return. The 33% from the backtest comes from the few numbers that had a lot of hits in the sample. The odds are better than even that one number will come up 10 or more times in 200 spins. Your technique will pick this number to bet on in the future, and project that it will come up 5% of the time for an 80% profit (0.05 x 36 - 1). The actual return on betting this number going forward is 3%.Your backtesting misleads you in two ways. First it gets you to pick the wrong strategy, the correct strategy is to bet on all numbers that came up 7 or more times in the 200. That has an expected return going forward of 1%. Second, it wildly overestimates the success your strategy will enjoy in the future.The antidote to this effect is to fit your model on one set of data, then test it on another, preferably data from a period later than the fitting period. Or test your model dynamically, updating the parameters and test each day based on the parameters you would have known at that time.Even that does not eliminate the problem. There may be unconsious biases in model selection and fitting based on what you know about all the data. This is one of several reasons that backtesting results have to be viewed with caution, especially if they are not computed by a careful, honest and experienced analyst.

Aaron, thanks for the extensive and clear explanation! You rock, man!

That's a great explanation Aaron, thanks!This is where the question was raised