hi, since Elie and David agreed we start a new thread here, I am copy-pasting below a couple of messages from the thread "Similarity Reduction":reza<< In other words I do not understand why it is assumed financial models are doomed to be inaccurate for EVER. >>I’ve often asked myself the same question … is this because Quantitative Finance is a younger discipline than Physics or is there a fundamental difference as Derman and Taleb affirm? One explanation in their favor comes from Behaviorial Finance: people who buy and sell stocks are not perfect and rational, and are subject to psychological pressures, modeling that would be really hard … while in physics we are observing objects, and psychology should not have anything to do with it …(except Schrodinger's cat, Wiener's friend ...) Elie<< while in physics we are observing objects, and psychology should not have anything to do with it …(except Schrodinger's cat, Wiener's friend ...) >>I disagree. The so-called reduction of the wave function, or the problem of measurement in QM, have nothing to do with the observer's consciousness. The reason such "paradoxes" happen in QM has to do with the "theory of knowledge" and by that I mean, the formal, or objective theory of knowledge (we can even call it the "logic" of knowledge) . QM is the purest and most "basic" of our theories, because it deals with the basic "objects" of our universe. (We can conduct experiments involving "single particles"). At this basic level, not only do you find that the notion of "object" and of the "properties" it may bear are ill-defined (would you call the "particle" your object? or the wave funtion? would you call Heisenberg's observables its properties? or would its property just be the pointer value in the measurement set-up?), but you also find that such notions as "experimental fact," "experimental observation," or even "experimental context," which are not specific to QM but constitutive of our whole epistemology, even the most classical one, are no longer to be understood in a prior, meta-physical or for that matter meta-epistemological framework, which you would THEN build you theory or your experimental expectations upon, but make sense only within the one and same process whereby you conduct your experiment.At the "basic" level of QM, epistemology is doomed with the problem of self-referentiality. The whole QM formalism is based on QM experimental outcome (the wave function gives only probabilities that such and such observable have such as such value), yet the QM experiment is itself a QM interaction, hence falls under the same formalism.The net result of all this is something the epistemologists call "the non-closure of the epistemological circle." Technically speaking, this means QM does not point to a classical framework where the "universe of possibilities" is settled once and for all, and Kolmogorov's probability theory can do its usual job. Rather, QM is a generalised probability theory, a meta-theory of probability if you like, which tells us what to expect when the observable phenomenon IS NO LONGER independent of the experimental context. Our classical world and classical epistemology will then appear as particular cases, or rather as sub-cases, of this generalised "theory of predictions."In a way, QM is not just a physical theory; neither is it a "paradoxical" physical theory which requires philosophical criticism in order to be understood. It is itself THE philosophy of physics. Now to come to markets, and to the question whether Quant Finance will ever be as accurate as physics, I think the problem is that we do not even know what science we are dealing with, when we talk of Quant Finance (At least we knew that QM was about particles, and basic observables). To my mind, Quant Finance is definitely not stochastic calculus, definitely not mathematical models or mathematical software like my own, definitely not trading programs, definitely not econo-physics, definitely not neural networks or fractal analogies, definitely not any complex, or intelligent, or self-adaptive, or self-learning, combination of all this. Nor is Quant Finance something irreducibly human or behavioural, so that biology or psychology should be the ultimate guides. At the basic level of QM, every experiment imposed its experimental context, in a way that was no longer "commutative" with another experimental context. This resulted in the peculiar "randomness" and paradoxically sounding probability formalism of QM. I think the problem is even "more basic" in Quant Finance, hence we should address it from an even higher and more general epistemological stance. "Experimental contexts" change every second in the markets, in a completely non commutative fashion. So I think we need to find the right epistemological ALTITUDE, wherefrom to look at what "we want to look at" and "what we want to expect" in Quant Finance, before addressing the questions it poses as a science. Here is a case (perhaps the only case), I think, where we need the philosophy of the science, before we can delimit the science.

Interesting, I tend to believe that Physics is an earlier node of the Quantitative Finance. Why? As a coincide, but in a form of rigid standpoint, both, the Physics and Quantitative finance had to elaborate tremendous amount of data with minimal models that merely scratched the surface. Physics during the 70s' started to develop new model (which we can assume model = assumption). Literally with new computer, power and tools could observe new dimensions, beyond the old models. The inevitable outcomes from those observations were equations and rigid laws.Many Physics during the 80s', such as Derman believed to overcome such distinctions between physics and quantitative finance and easily construct new models and measurement tools borrowed from the physics world into the quantitative finance realm. At least, they avoid the illusion of been able to structure rigid equations and laws before applying dept and skeptical observations in the financial markets. In these recent days, the traditional physics withstand a breakout gap of new "objects" with certain memories that dissipated doubts that cannot compare to fluid dynamic or quantum mechanics. Surprisingly, the source of the breakout comes from genome and molecular science. The ability to measure specific splicing variant and predict it expression during "wet test" required highly sophisticated computing power. Here we have new form of analysis of proteins and nucleic acids (DNA and RNA). DNA is present in almost all living cells and holds the genetic information necessary for the development of the entire organism. Therefore, the information in the nucleic acids contains "memory" (the ability to identify and attack foreign objects in the body is a good sample). DNA chips or Biochips, are poised to revolutionize the biotech industry by performing a vast number of parallel experiments in a single go. Currently, the technology and market of DNA chips are still in their infancy, but not far the day that the distinction between human memory to new QFM programs will shorten. Obviously, with minimal convincing effort, one may contradict any possible implementations of genome science in QF. Careful observations are very necessary. Nevertheless, shall we assimilate it to neural networks and constructs black boxes?Of course not, my point is using these memories by pure back testing (wet tests) the data and then performs a flexible model. Thus, the program will learn old human vagaries and will present such and such probabilities under a certain market conditions. In my humble opinion, from practical and philosophic viewpoint, the sequence of new paradigms in the financial markets might spread into new and outstanding sectors, beyond physics, human psychology, computer science, and roads engineering. In other words, smarter and more flexible agents in trading and modeling software will help the practitioner in his favorite attitude and vagaries and narrow the assumptions making of such and such probabilities and this or this simulation process. The best parallel description relates to genome medicine that interrogates the very specific organism of a very specific human body. Therefore, it possible to step beyond the fluid dynamics and QM, although the crucial aspects that intertwine this smart agent with such memory still, how we do it?David

Aaron Brown said:The goal of quantitative finance is not exact prediction. If we knew the stock price of MSFT ten years from today, there would be no more need for finance. Everything would be discounted at the risk-free rate.The goal is to decompose financial numbers into a pure random and pure deterministic component. In the Black-Scholes world, for example, the stock price changes are completely random and the option price at any point in time is completely deterministic.In physics as a practical discipline (i.e. engineering) the goal is the same. We compute the load a given bridge design can support, then triple it to account for all the random factors. We compute the orbits of planets, but use a random model to predict coin flips and gas pressure. Someone may believe that everything is deterministic or that there is essential randomness, but practical people use the equation that works best for the problem at hand.I think behavioral finance has the same goal. It sets about achieving it with a more complex model of human behavior because that may lead to better financial models, not because it cares about the internal psychological states of human beings. I do not predict success for the school, but not because it's wrong (its models of human behavior are clearly much more accurate than standard finance), but because the more complex models lead to impractical numbers of free parameters.Practical finance will always be less accurate than theoretical physics, but that's because practice is less accurate than theory rather than finance is less accurate than physics. I can predict an FX forward exchange rate (given the spot and both countries' interest rates) more accurately than a physicist can predict how long a light bulb will last under constant conditions. I can compute a Black-Scholes option price to more decimal places than any physical constant is known. *******************************************************Aaron, that’s a good point, which makes me think perhaps this is a observation/ measurement and precision issue.In classical physics we can fairly easily measure distance, time, speed … and therefore the laws work well in their applicability domain … while we have a hard time observing and measuring infinitely small events with precision. Now in finance the Black-Scholes model gives results that are valid within a certain precision level but that precision is not enough for us unless the level of liquidity goes infinitely higher … a little bit like the Central Limit Theorem which is only asymptotically valid … so perhaps Quantitative Finance is more like Statistical Physics?hope I make sense, I am not totally sure what this all means …

Since in these kind of philosophical issues the amount of different opinions tend to diverge I will try to keep mine as short as possible.when I think of financial models of being predictive I mean predicting probabilities, not actual outcomes. In other words, for me a financial model would be PERFECT if it is able to generate financial behavior probabilistically indistinguishable from real time series. Do you remember some of the quizzes at Wilmott? What we need is model generating good fake series! That is, we need our models to pass a financial Turing test.I do not know if this 'modest' requirement is achievable, but at the same time I am not convinced that there are reasons to believe this is impossible (even if humans are involved (at least if we deal with a large quantity of individuals))Quoting Aaron againPractical finance will always be less accurate than theoretical physics, but that's because practice is less accurate than theory rather than finance is less accurate than physics. I can predict an FX forward exchange rate (given the spot and both countries' interest rates) more accurately than a physicist can predict how long a light bulb will last under constant conditions. I can compute a Black-Scholes option price to more decimal places than any physical constant is known. >>perhaps physics cannot predict how long a light bulb will last under constant conditions but it can predict the fading rate behavior of 10^10 of such light bulbs with notable accuracy!with this I want to illustrate that if we want to obtain some accurate laws in finance perhaps it's good to start thinking of the right questions. Even if these questions are not exactly the ones we would like to be answered from a practical point of view.or quoting Elie Ayache I think we need to find the right epistemological ALTITUDE, wherefrom to look at what "we want to look at" and "what we want to expect" in Quant Finance, before addressing the questions it poses as a science >>Cheers,Julianps: sorry for not being brief enough!

In reference to Aaron last thread in Similarity Reduction.I tend to agree with Aaron that the goals of quantitative and behavioral finance are not to predict a future outcome. However, it highly unlike that the goal is to decompose financial numbers into pure random and pure deterministic components, at least, in behavioral finance. Actually, the behavioral finance attempts to find deterministic nodes in a randomness tree. The approach to manifest the deterministic nodes which could be expanded through Taylor method (see Lo & Mackinley as well Taleb). The evidences of nonrandom events appear in many forms of relatively minor time segments, such as, crossing a barrier, ex-dividend day, and a combination of anomaly and investors sentiment (see Shleifer on "Inefficient Markets"). Furthermore, there are plenty of irrational events in the markets that could be explained as deterministic approach from investor's viewpoints, tax advantages, squeezes, and the last horrified terrorist attack combined with the Put options and shorting indexes. Sometimes, the pricing process of certain instruments is irrespective. Whenever, it more likely to call these irrational events as "vagaries" from a financial vogue. Random and probabilities are never to be pure, since it possible to identify nonrandom variables in inefficient markets. On the other hand, I do agree that behavioral finance models are complex, because it stills in its infancy. Efforts to simplify the complex models of the behavioral finance are on the way. David

Hi David,I agree with you in thatRandom and probabilities are never to be pure, since it possible to identify nonrandom variables in inefficient markets. David >>but from the modelization point of view the separation between random and non-random components is not necessary since you can always consider a deterministic variable as random one with a Dirac-like distribution, so at the end a 'mixed' variable is just a random variable.J.

… so guys, what do you think about the idea that Quantitative Finance is to be compared to Statistical Physics and will become exact only asymptotically, meaning only if the number of transactions becomes infinitely large (like the Central Limit Theorem)? …

… so guys, what do you think about the idea that Quantitative Finance is to be compared to Statistical Physics and will become exact only asymptotically, meaning only if the number of transactions becomes infinitely large (like the Central Limit Theorem)? … >>I think the more the liquidity the more easiest to guess the correct statistical behaviour.So I do not know exactly what do you mean by "become exact only asymptotically", In QM you can describe the statistical behavior of fotons in a two slit interference experiment both with large or low quantities of fotons. The only difference is that with large quantities you can 'see' the interference fringes much easily.J.

… so guys, what do you think about the idea that Quantitative Finance is to be compared to Statistical Physics and will become exact only asymptotically, meaning only if the number of transactions becomes infinitely large (like the Central Limit Theorem)? … >>I think the key difference is that we don't wait around for enough observations to make our theories accurate enough, we can change financial practice. This is what makes finance an inherently practical field. We figure out theories and test them. If they fail, that implies a money-making opportunity (for any financial theory I am interested in, anyway). We try it and either get infinitely rich, discover our error or change market practice so it conforms to the theory.In the last 25 years we have seen a huge increase in the sophistication of financial products. All of these products were designed by quants, all were introduced in order to make money. The market prices of these products provide the raw data for a new generation of theories.I don't think this process asymptotically approaches completely efficient, completely understood markets. I think there is a limit to how much effort people will expend (although that limit goes up as financial markets increase in size and transactional efficiency). Also, I think there is enough change in the economy that there will always be new wrinkles.

re: philosophy and finance - how closely ought the theory relate to the practice? in economics, for example, the theory often bears very little practical resemblance to the observed reality; instead, it provides polarised depictions of an idealised, abstract world which theoerists and practitioners know will never be evident ouwith a textbook. only particular aspects of the theory can be seen to manifest themselves in the real world. so how should quant finance work? there must surely be a question re: whether to have generalisable or specific models.

I have no patience for theoretical finance. It may be valuable, but it's not finance. Theory is definitely useful, but if you're not willing to bet your own real money on your work, you're an economist in my book. When finance loses its connection to making money, and starts telling people how they should behave instead of getting rich exploiting misbehavior, it loses its reason for being.We all know the ultimate theoretical answer, everyone should love everyone, give according to their abilities and take according to their needs. Until that Peaceable Kingdom, the goal is to figure out what people actually do.

ScilabGuru said in the "The discrete and the continuous" thread:What I like in BS that1. It is selfnoncontradictory (to the best of my knowlege)2. It allows simulation of an ideal market which qualitatively is similar to reality3. In many cases quantitatively it gives good resultsActually, this is what means "good model" >>You're right about the "good model." Actually I was planning to elaborate the difference that I see between the way the physicist thinks of a "good model" and the way the financial engineer "should" think of a "good model" in the "Philosophy, Physics, and Finance" thread.Physicists have a tendency (at least those who haven't completely read through the philosophical conclusions of Quantum Mechanics) to seek an ideal model that would, in the ideal end, account for the physical world in a completely autonomous and objective way. Their hope is to find the right ultimate parameters, or - for those among them inclined to self-learning processes - the right way the system itself can look for, and adapt, its own parameters, so that an all-encompassing model is finally written, that would take care of itself, of the world, and of its representation of the world, without there being any place left or any job left for the "subject," or indeed for any agent that would so much as make sense of the whole idea.This hyper-objective phantasy has been labelled, eloquently enough, "the view from nowhere" by Thomas Nagel. If you read through the "Philosophy, Physics, and Finance" thread you will recognise the very same dream surviving here among us financial engineers. The master word is "calibration." It produces the following sequence of conditionals. IF I can calibrate my model to such and such empirical data, say the prices of vanilla options, THEN the model will predict such and such value for such and such exotic. And IF empirical data later diverges from my prediction, THEN I will enlarge my model, include the exotics in the calibration procedure, and predict the rest. This goes on and on, until we reach a super-model that is calibrated to all market prices. But then, what would we need it for? (Not mentioning that we will have to re-calibrate the next day!). Or else, we just decide to take the ultimate, all-encompassing, probability distribution that the model finally implies for granted, and actually expect the market, the next day, to probabilistically behave the way we think.Notice how the chain of conditionals has progressively eliminated the "subject" from the picture. The trader is expected to step outside the trading room, and leave it in the model's hands, the day this ultimate model will be in place. This is not my idea of a "good model." As the number of parameters multiplies and quantitative models grow more and more complex - this much, we know for sure, and there is unfortunately "no way round" technological progress - I expect, on the contrary, that the trader will take a greater and greater part in the process, and that the complexity of the models will require, ever more, the "final" and single decision of the trader.As you say, a good model should be non-contradictory and should allow SIMULATION. This is the key word, to my mind. Models are just fictional, yet consistent, projections, and the chain of conditionals should, as a matter of fact, go exactly the opposite way. IF the world is like Black-Scholes say it is, THEN the option prices will be such and such. IF the world is like my favourite smile model say it is, then the exotic prices will be such and such. The dreaming physicist wanted to encircle the world with his chain of IFs, and dispose of the trader in the ideal end. Whereas what we are here contemplating is a completely open world, with the trader standing firmly at the centre, and the conditionals actually radiating from the centre. But will the radiating fictions ever tell the trader what to do in the end? Will the trader learn, from the Black-Scholes fiction, or the stochastic volatility fiction, or the jump-diffusion fiction, what actual trading decision to make in reality? I think he will, but I think there is no way we can model THAT. Action is the other side of representation in a trader's mind, and the ultimate "model" in Finance, or in other words, the ultimate science we are all looking for in Quant Finance - and of which many are still debating whether it will ever be as "accurate" as Physics - cannot dispense with either side of the coin. How big a part the trader plays in the model, should actually be a requirement of a "good model" in Quant Finance.In a way, we all go to the movies, or read novels, TO KNOW what to do, and how to act, in reality. And I like my movies and my novels to be as exacting and as rigorous as the best science I know, however fictional and other-wordly they may be. Science and its models give us finished objects to reflect upon; they don't unveil reality. Now the reason why it might seem that the situation I've just described is specific to Finance and irrelevant to Physics, is that the trader is above all a man of action, while the physicist is usually perceived as just a passive spectator of the universe. Such a discriminatory view would amount to ignoring that there is one big and true action going on everyday in Physics and, for that matter, in all the natural sciences, that of UNDERSTANDING the world.

NumberSix, I liked your last message very much. I'd like to add some ideas from my experience. The general problem of Statistics/Econometric/finance is that ALWAYS there is no enough data.In Physics one can make a billion experiments to measure say gravitaion constant, and to get result with any accuracy. There is noproblem to fit curves/processes in Physics. You are playing with different models to achieve the better and better results. Why?Because tomorrow you can repeat the experiment and you will get a process with the same properies. In finance this is not only impossible, it is forbidden from some point of view! (It is related to your "calibration story", numbersix.) There is a notion of "overfitting" that is not only to take too muchparameters to fit a curve but much more general, philosopical and complicated stuff. Assume you have an experimental curve (prices) return of which you want to predict. Naive approach is to try to fitthe whole curve. Then you can get a brilliant fit "in sample", butnobody said that the future unknown values of the curve (out of sample) will lie on your predicted curve. The more sofisticated and correct approach isto split curve into two parts "the past" and "the future". You fitthe past part measuring performance on the "future" part. Then you canget more robust result. But what is going on if you tried 1000 different models? One of these model suddently can give you very goodfit on the "future" part of your curve. But since you tried to fitthe curve 1000 times this model dose not worth nothing anymore, because you could get a good fit just occasionally. The conclusion is that a researcher should:1. Do not play with the same data too much2. make protocol of the experiment3. compare models not only with respect to the quality of the fit butalso with respect to complexity. The simpler model is better. (Occamrazor, Bayesian approach)4. estimate statistical significance of the results obtainedThis makes life rather difficultScilably yours

Does anyone have an opinion as to why there are relatively few Economic PhDs in the financial quant field relative to physicysts, etc.? In the US, economics PhDs are sufficiently quantitative, and the familiarity with the broad subject matter, as well as nonstationary and limited datasets should be an advantage. Yet clearly the market is saying that "quant" jobs are better filled by the hard scientists. What's the driver here?

One game theorist I know has once told me that the difference between Economics and Finance is that Finance has finally "made it" as an engineering science. This what explained its growth to his eyes, and the appeal that the hard scientists feel for it.