Hi,I need to compute MSE (mean square error) of models relative to observed data. We compute MSE in order to compare models. However, in the literature, researcher do not test whether their results are significantly different accross models. Which statistic would you use?Tx in advance

-- competition is often a good thing, but there may be a reason for not comparing some models to each other. A lot of papers develop models to explore a particular phenomena. They're not so concenered with the fact that they've developed a good model they just want to show that factor-x has predictive power for equity returns or interest rates, or whatever (maybe factor-x can be combined with factor-y to build a "better" model, but they don't want to distract you). ...In other cases, a comparison makes sense (e.g. what is better a square-root rule, a linear-rule, or something in between), but you what point is there in comparing an interest rate model to an equity model (maybe some, but it is too complicated by other, non-modeled factors).-- if you were going to compare models, why not bypass the elementary statistics and go with some risk-reward framework (i.e. how much money would I make using your model). This is finance after all. I've heard quite a few academics say this is a good idea, but you rarely see it in practice.

Ok Mike, but imagine that the good measure is MSE (and the framework might be smthing else than finance...). Then, which statistic would you use in order to compare the MSE measures ?

I'm afraid you always need to use MLE, and MSE is rarely equivalent to MLE. Anyone know why??

I am afraid N, but you don't...MLE requires the assumption that errors are Gaussian.MSE (mean square error) or MAE (mean absolue error) are comparison measures. I don't want to optimize in order to get the minimal MSE or MAE. I just want to compare the results...Nobody has an idea ?

QuoteOriginally posted by: SPAAGGOk Mike, but imagine that the good measure is MSE (and the framework might be smthing else than finance...). Then, which statistic would you use in order to compare the MSE measures ?If we are assuming that MSE is the correct measure then what it the problem? More error is worse (given the same data set). ...I'm probably missing something. Where is the contradiction in comparing the MSE of two models?

There is no contraditions... but I want to make inference with my results. For example say that, at the 95% level, MSE of models 1 and 2 are differentsmthing like that

chi-square distribution? ...not sure, just a guess.

QuoteOriginally posted by: SPAAGGHi,I need to compute MSE (mean square error) of models relative to observed data. We compute MSE in order to compare models. However, in the literature, researcher do not test whether their results are significantly different accross models. Which statistic would you use?Tx in advanceIf your primary interest is in comparing the accuracy of your models you can look into the Diebold-Mariano test statistic , in Splus-Finmetrics computes this (i think), Details on this can be had in their paper Comparing predictive accuracyhth

Quantie,Very interesting paper. Do you know if the same is available in R?Thanks

QuoteOriginally posted by: nonoQuantie,Very interesting paper. Do you know if the same is available in R?ThanksNo I haven't seen it but it should be easy to cookup.There is some rats code online if you search!hth