Well, we just disagree here. Or you are simply trolling.

I don't think you can sensibly think about (or invest in!) the market without having some idea of what the equity risk premium and volatility have been historically and that requires measuring them. Basic notions like autocorrelation were important in understanding that securities markets are, in some sense, "close" to random walks. Another well-established insight is that broad market equity return distributions, for short return periods like a day, have "wide tails" relative to Gaussians. I guess you don't accept that because you can't measure kurtosis?

I have to ask. I assumed you were in finance. If so, what do you do all day?

  

I don't think you can sensibly think about (or invest in!) the market without having some idea of what the equity risk premium and volatility have been historically and that requires measuring them. Basic notions like autocorrelation were important in understanding that securities markets are, in some sense, "close" to random walks. Another well-established insight is that broad market equity return distributions, for short return periods like a day, have "wide tails" relative to Gaussians. I guess you don't accept that because you can't measure kurtosis?  

What I think is still a subjective probability.

What did I do so far? Quite a lot: university, various quant positions, management of people, senior management in banks, my own investments, property development. Nothing to be ashamed.

But you know - it is dangerous to talk about your interlocutor. You're too close to the Cretan paradox.

I forgot trading equity derivatives. But it was long time ago and very boring as far as I remember.

Is there a field of applied statistics where things could go as wrong as they did in mathematical finance? Distributions, fat tails, kurtosis… If there’s a market crash, are you considering with what probability you actually went bankrupt? Just wondering if only my brain rebels against thinking about it in these terms.

Is there a field of applied statistics where things could go as wrong as they did in mathematical finance? Distributions, fat tails, kurtosis… If there’s a market crash, are you considering with what probability you actually went bankrupt? Just wondering if only my brain rebels against thinking about it in these terms.

There's a time to think about probabilities, and a time to think about worst cases. And part of that is whether it's your money or someone else's, and whether others are in the same boat or it's just you. And don't forget to allow for movie deals and speaking engagements.

Is there a field of applied statistics where things could go as wrong as they did in mathematical finance? 

Of course not, and we know the exact reason: electrons are identical, stones are almost identical, bacteria are similar, and even hamsters are not that different. Even more - humans are similar as long as they eat, sleep, work, reproduce, spend money and do all the things in which they are not very different from hamsters. But when human intelligence comes into play, everything changes, we are no longer so similar. The assumptions of statistics are not fulfilled, and 'garbage in' means 'garbage out'.

Is there a field of applied statistics where things could go as wrong as they did in mathematical finance? Distributions, fat tails, kurtosis… If there’s a market crash, are you considering with what probability you actually went bankrupt? Just wondering if only my brain rebels against thinking about it in these terms.

There's a time to think about probabilities, and a time to think about worst cases. And part of that is whether it's your money or someone else's, and whether others are in the same boat or it's just you. And don't forget to allow for movie deals and speaking engagements.

How come? Those probabilities don’t indicate any specific time. You learn it the hard way when it’s too late. Until if happens they only tell you how much you should shake in your pants.

Is there a field of applied statistics where things could go as wrong as they did in mathematical finance? 

Of course not, and we know the exact reason: electrons are identical, stones are almost identical, bacteria are similar, and even hamsters are not that different. Even more - humans are similar as long as they eat, sleep, work, reproduce, spend money and do all the things in which they are not very different from hamsters. But when human intelligence comes into play, everything changes, we are no longer so similar. The assumptions of statistics are not fulfilled, and 'garbage in' means 'garbage out'.

So why aren’t you guys using Bayesian statistics to begin with?

Isn't 'Bayesian' a synonym of 'subjective'?

Depends on your prior

Is there a field of applied statistics where things could go as wrong as they did in mathematical finance? Distributions, fat tails, kurtosis… If there’s a market crash, are you considering with what probability you actually went bankrupt? Just wondering if only my brain rebels against thinking about it in these terms.

There's a time to think about probabilities, and a time to think about worst cases. And part of that is whether it's your money or someone else's, and whether others are in the same boat or it's just you. And don't forget to allow for movie deals and speaking engagements.

How come? Those probabilities don’t indicate any specific time. You learn it the hard way when it’s too late. Until if happens they only tell you   how much you should shake in your pants.

Should I go crosstown first or downtown? I can take advantage of the red light but there'll probably be other red lights on the way. Oh, hang on, that school bus doesn't seem to be braking.

FWIW I'm agreeing with you.

But when human intelligence comes into play, everything changes, we are no longer so similar. The assumptions of statistics are not fulfilled, and 'garbage in' means 'garbage out'.

Is it necessarily so?

I mean, in the confined probability spaces of games of chance, optimal strategies (usually, "don't play in the first place") can, to a large degree, be determined. Some fool might think he's on a hot streak at the craps table, and some other fools might believe him, but in the end they are wrong (making usual assumptions: game not rigged, most of all). They may get lucky and win anyway, but they are wrong.

Now, craps results are purely determined by theoretically purely random events; it doesn't really matter if the players have oddball ideas about luck or fate. Games like poker add an element of strategy/deception that make it more difficult to develop optimal strategies, and that make "reading" other players an important skill in the game.

But, ideally, players cannot be "read," and ideal play has optimal strategies, albeit strategies that involve randomizing toward the general goal of maximizing the uncertainty of opponents. I think AI poker players already beat the very best human players, probably (in my opinion) just because they do better at hitting the right probabilities for random actions than humans do. Compare that to chess, which is completely deterministic, but which took AI decades to finally become better than the best human players.

Financial markets involve immensely more information than games of chance, but if you can process that efficiently -- which I think is more akin to the problem of solving chess than of solving poker -- why wouldn't it boil down to being very similar to a game of pure chance, where optimal strategies can, to a significant degree, be determined?

Of course there are aspects of human behavior that enter into financial markets, but a lot of those are effectively random: will there be a sudden wave of people going vegan around the world? Will electric vehicles fall out of favor? These sorts of things are effectively discordant with the goals of someone trying to understand financial markets, and not adversarial like the behavior of opponents in a poker game ... the significance of which is, you can generally plug in a random factor -- like the die roll in craps -- and, if you have chosen well, be done, without worrying that some adversary will figure out what you're doing and how to use that knowledge to confound your efforts.

(Are we any closer to giving original poster Peniel a problem to solve?)

(Are we any closer to giving original poster Peniel a problem to solve?)

Well, if he doesn't like the original one I gave, clearly the proper use of statistics in finance is still an unsolved one!

Well, here's a problem for Peniel, and one that sort of dovetails with Alan's project: develop a methodology for finding hidden factors in financial prices.

As background for this, look at the Fama-French Five Factor Equity Model, the five factors being beta, size, value, profitability, and investment.

But these five factors came about as a hodgepodge, as far as I can tell: they had existing models that left a lot of uncertainty, so they apparently did "what if we controlled for x?" experiments and came up with some things that seemed like they removed significant amounts of uncertainty.

I think the most interesting things, however, come about not because someone introduces a consideration (which almost of necessity means that he believed there was a possibility that the consideration would be significant), but rather because it is clear that something is going on and it doesn't even have a name. Newton deciding that the force of gravity must be proportional to the inverse of distance squared based on Kepler's Laws, for example.

Given that we can now do big data things, how do you go about finding the angels and the demons -- creatures purely of myth -- that move financial prices?

I will focus on Alan's problem (@Alan, I've sent you an email).
I'll leave the angels and demons to you guys, I am clearly not ready nor properly trained for this quest yet.