Algoritmic trading utilizes data mining techniques to extract relationships in data in order to exploit them. Taking into account the vast number of firms performing data mining, we can conclude that any relationship extracted from data should be quickly exploited by too many exploiters (efficient market). So all algoritmic trading strategies should be shortly lived since the profits should quickly evaporate. So why all these banks put so much emphasis on algoritmic trading if its profitability is not so good? Is it a new irrational fashion? Or I completely missed the point?

market efficiency is a theory, reality is differentthere isn't just a few well-known ways to analyze data but on the contrary an infinity ..some techniques can definitely generate sustainable profits, many others don't !

To add to g000RRRe's point, you'd be surprised (horrified perhaps, like me) by the amount of trading that goes on using ideas that haven't been tested or even ideas that have been tested and shown to not work.P

This view assumes 1) that a trading strategy is a passive extractor of exogenously-supplied excess returns -- i.e., that the trader merely harvests preexisting inefficiencies; 2) that all traders are equivalent (deterministically or stochastically) in their ability to recognize patterns and execute trades against them; 3) that traders (or their employers) are rational. None of the assumptions is right and that creates the potential for persistent profits for some traders (and persistent losses for others).First, traders can view themselves and the market as part of a controllable dynamical system. If some of the trader's orders induce inefficient behavior (e.g., stop-hunting, bluffing other traders, or attracting excess capital to an instrument), then the trader can create sustained profits. Yes, this does seem like market manipulation but it's also the reason why many traders see poker (not other casino games of pure chance) as a better analog for trading. The broader issue is that neither the price pattern nor the trader are exogenous to the system. Rather each trader's behavior affects the price pattern and the trading-induced price pattern affects other traders' behavior. Knowing that, it makes sense and profit to play a deeper game. (As an aside, nonlinear chaotic dynamical systems have the paradoxical property of being both less predictable and MORE controllable than linear dynamical systems.)Second, although the addition of more traders may subdivide the total profitability of a given strategy, that doesn't imply that some traders won't continue to make consistent profits (perhaps even growing profits). Consider an observable arbitrage opportunity and the dynamics by which traders discover the pattern and trade it. The first trader to find the pattern will have the pattern all to themselves and will profit for a time. But what happens when a second (or additional) trader finds the pattern? Will they equally share in the profits with incumbent trader(s)? The answer is no. Traders with faster decision-making and execution platforms will take more of the pattern's profit than traders with slower execution platforms. What's interesting is that traders with lagging execution may actually help boost the profitability of traders with leading execution. One can even imagine 110%:-10% split of the profits -- slow traders may bleed capital to fast traders. Moreover, for some broad categories of trading patterns (including momentum trading) the growth in the amount of capital applied to the pattern strengthens the pattern -- i.e. the more traders that trade on momentum, the more momentum one will see. The result is a persistent pattern that attracts new traders who, if they have a slower decision-making and execution, will bleed capital to faster traders. (Traders will also systematically assort themselves on transactional overhead and risk-tolerance in front-running any pattern.)Third, (and as Paul describes) traders and IBs are not rational in the sense that they don't maximize long-term profits, don't minimize long-term risks, don't use sound logical deductive/inductive processes for trading. The incentives at many IBs favor risk-taking above profit making. One recent article in the WSJ noted that compensation at the big IBs rose 9% last year even as marketcaps on these IBs dropped significantly -- IB employees received an average of $350,000 in return for destroying $275,000 in market cap per IB employee. One could even imagine that the ultimate strategy of any IB is to stir up the hornets nest of the markets because the IBs make the most profits when the paper is flying fast and furious (which takes us back to strategy 1 of trading to induce inefficiencies).

QuoteOriginally posted by: AlanFordAlgoritmic trading utilizes data mining techniques to extract relationships in data in order to exploit them. Taking into account the vast number of firms performing data mining, we can conclude that any relationship extracted from data should be quickly exploited by too many exploiters (efficient market). So all algoritmic trading strategies should be shortly lived since the profits should quickly evaporate. So why all these banks put so much emphasis on algoritmic trading if its profitability is not so good? Is it a new irrational fashion? Or I completely missed the point? I think that the sole existence of a consistent picture of algorithm trading is already an evidence that it does works to some extent. Personally I don't think that its point is in maximize returns, but it is in minimizing costs and risk. There is a economy in scale in using computers. As an example, suppose there is a chartist that trades according to some logical rules which seems to work most part of the time. One chartist can only analize 8 (maybe 10) stocks or more at a time, while a domestic computer, assuming a strategy based on simple calculations (eg. pairs trading), can do it for 8.000+. Therefore, there is massive economy of scale if you assume that SOME trading rules work since you'll not be paying salary for 100 chartists, but for just a few quants. You can even assume that on average the chartists do better than algo trading, but, since there is less cost, the profit of algo trading is still more atractive.Now, if you assume that pure logicals rules don't work, then its a whole different discussion of wheter intuition exists and chartists really do have something that cannot be translated into codes or if it is just rational rules with bits of randomness blurring the image.

Theoretically, you should add some new information. By using a price input or time series that hasn't been used before, or using a popular one in a different way.And since nobody is suicidal, the more efficient it gets, the larger it can grow to be - like the auto industry or the computer industry. If some idiot showed up selling computers for 10 cents and lost $150 billion, all the computer makers would be hurt by the volatility. But as long as it is an even competition, and nobody is dumb enough to compete prices down to where he would make less than a fair risk-adjusted rate of return, everyone can cover his labor costs. Since you expect nobody will compete it down to prices where he can't cover his labor and capital costs - and the confidence in this expectation increases as the industry matures and becomes more efficient - it is safe to allocate huge amounts of labor and capital to it, and compete for tiny edges. Reality is somewhat less perfect than theory, but these basic principles are the strongest signal in the profitability series.As algorithmic trading becomes less profitable relative to other activities - like protein modeling - labor is allocated away from algorithmic trading, and into protein modeling, so that the two remain equally profitable.

I know this is a little off topic, but is anyone else worried about the wave of algorithmic trading/backtesting tools currently offered by discount brokers in the US? It is pretty much received wisdom that program trading by the big financial firms exacerbated the previous equity market crashes. If the "little guys" start doing this won't this make the next crash even worse?

QuoteOriginally posted by: DavidJNprogram trading by the big financial firms exacerbated the previous equity market crashesI disagree that one crash - 1987 - merits an "es" on the end of crashes. That wasn't so much program trading, as a mirage about leverage and transaction cost that formed with the introduction of the S&P future on the CME. Anyway, the brokers offer the tools to high-frequency traders. So I think they will realize on the ninth or tenth trade before lunchtime, if their own trades are being fed back as signals to trigger their own trades, and try a counter-trend strategy.

QuoteBeware of quants? Beware of hedge fund geeks bearing greeks? Some models don't work therefore ALL models don't work? It is curious how fear and hype lead to unjustified generalizations. Semantic pigeon holing is a common response to the unfamiliar. Presumably that's why some think quants are in a quandary, derivatives are dangerous and leverage is lunacy. Black box trading systems are "secret" and ALL the same(!) so better to stay with "human" methods of making money? There are many more bad human investors out there than bad quant models, so beware of the QUALS as well as the QUANTS.Investing successfully is hard. It makes sense to use all available tools. A systematic, replicable investment process using qualitative AND quantitative analysis is surely the foundation of any successful hedge fund, though how they weight the two varies. The simple fact is there are GOOD pricing and trading models around and there are BAD ones. It usually takes bear markets and volatility to show which is which. But whether the models produce positive or negative alpha is entirely up to human inputs and human specification. Garbage in, garbage out or quality in, quality out.I think investors should beware of everything. Considering the non-quant problems and dire risk management policies on display recently, the faith in the value of human discretion seems ironic. Sure there are plenty of poorly designed and badly tested quant trading systems out there as there are delusional pricing models, but that does not preclude the existence of quality, robust products.A computer making the trading decisions rather than a human does NOT mean an increase in systemic risk or a decrease in the persistence of a good strategy. It just puts the emphasis on ensuring the computer is making decisions in a different way to other computers and turning away investors who require transparency. One of the biggest risks for any hedge fund is that its human assets walk out of the door each day. Computers can monitor the world 24/7/365 and the trading room is their home. Their loyalty is absolute. Computers can absorb and react to information on 100,000 securities instantly unlike a human trader. They can analyze new data, have the order in and executed before a human brain has even noticed the information.Computers don't think they are a genius when they fluke a lucky trade. They don't take lunch or vacations. They don't quit and try to set up their own fund with proprietary information. They don't have clandestine meetings with competitors. They don't complain about colleagues, clients and bonuses. They don't get sick or crash their Porsches. They don't lose interest after the IPO. And once the programming and testing is done you have eliminated key-person risk.There is a lot in favor of purely systematic strategies IF they are good.The main distinction comes down to whether the human decides, or the computer programmed by humans decides. But is that really a distinction? If a systematic trading model needs adjusting to "new" phenomena then it wasn't properly tested in the first place. Why the fuss about "ALL" quant funds? It probably comes down to the fear of the unknown and hatred of opacity.Discretionary investors can be reasonably open about how they pick stocks since the "edge" is the skill in implementing the strategy. Good systematic strategy developers cannot be so open since;1) the edge is the strategy2) no-one outside quant land will understand3) those inside quant land will steal it leading to the inefficiency disappearing and trade crowding problems.Any successful investment strategy needs a robust decision-making framework and elimination of emotions. The best way is a division of labor between humans who are good at gathering data and machines that are good at processing that data in the way a human asks them to do. The more short term the trading the more useful artificial intelligence is going to be.It makes sense that PROVIDED the algorithm has been put together competently, to ask the computer to trade IF time is of the essence. High frequency trading is very dependent on low latency and incorporating a human override slows thing down. With high frequency the speed of execution and reduction of slippage often IS the primary profit driver. I've heard so many times that quant investing will replace humans but conversely I am hearing, yet again, that "This is the end of quant"!Both are wrong. There is nothing new about bad quantitative models running into problems. There have been several analyses of the so-called “Quant Meltdown” as if it was the first time this had occurred. This is similar to the portfolio insurance implosion of 1987, the mortgage-backed securities pricing "models" of 1994 or LTCM's option pricing "geniuses" in 1998.Just as there are good and bad human stock-pickers, there are good and bad human quants. Show me a human based strategy that hasn't also run into problems at some time. Just because public domain quant strategies using the same methodologies will identify the same stories and opportunities does not invalidate other proprietary methods. The models that ran into trouble – either;1)find pairs of stocks historically cointegrated and take the other side when they are X sigma apart or2)throw every fundamental and technical variable you can think of into the hopper and data mine for what patterns worked in the past - are now very crowded.Apart from some now very large hedge funds (a few good, many bad), there were investment bank proprietary desks heavily in the statistical arbitrage and factor model strategies. Some multistrategy hedge funds that couldn't unwind illiquid credit instruments were forced to unwind what was liquid to meet margin calls. The risk of comingling liquid and illiquid securities is one reason why I think their will always be demand for single-strategy funds.The situation was also exacerbated by some of the 130/30 crowd panicking when their shorts began to tick up on all the short covering. I wonder how many of them knew beforehand that short positions get bigger as you lose money. I wouldn't be surprised if some of the less experienced 130/30 entrants were temporarily more like 120/40 or even 110/50 in early August.Models are only as good as the assumptions humans give them and the programmer's representation of reality. Unfortunately reality is rather complicated. To put it mildly, the facts have not been kind to the theories.If you code up some C++ or C# and tell the computer that we live in a nice "normal", "standard" world of rational entities that spend their days maximizing their utility and immediately changing prices accurately to new information then you will run into trouble.The computer only knows things that YOU choose to let the computer know about. If you lose money beyond statistical expectation then that is a human error, not computer error. Try typing =850*77.1 into Microsoft Excel 2007. Is it a human design error or the computer's fault if you get 100,000 instead of the correct 65,535? No computer would have "valued" Facebook at $15 billion either. Computers are just a tool.Humans design "discretionary" investment strategies and they design "systematic" investment strategies. If they are good or bad it’s all up to human ingenuity or human stupidity.Whether they data mine the past or test hypotheses of the future it’s all up to human skill. Computers are good at information processing but can only analyze the data they are given in the way humans designate.Quantitative risk management is only possible based on the factors input to the system; if the machine has a blind-spot to a new factor, there will be errors often of a non-linear order of magnitude.Computers are simple creatures; if you only tell them about bell-curves and the "rarity" of 3 sigma moves then they are not going to perform very well when 25 sigma moves come along. There are no axioms or proofs in real world markets. Asset classes don't read textbooks.An IQ test can be coached but the market is an IQ test where the questions and answers change while you are taking it. If you assume randomness and rationality in a deterministic, chaotic process like the markets, then your models are going to be wrong.As we have seen recently, everything is connected, so therefore models that rely on independence are headed for trouble. A good model is one that provides;- a persistent trading or pricing edge, - can cope with a non-linear, dependent, varying factor world and - an underlying theory and equations that have NEVER been published. Some quants like to use something called "stochastic calculus" which is useful for many things but certainly NOT financial modelling. As simple tools computers are not good at complex event analysis because most programming hasn't focused on that area. Unless its human owner has informed it that most CDO pricing is wrong, that if Andy defaults then the chance of Bob and Chuck also defaulting is much higher than the credit models "assume", and that there are a bunch of other people out there running very similar equity mean reversion programs, then the model won't pick up that maybe it should change things. It just follows orders. If quants neglect to inform their computers that if a weaker player with a similar model is forced to unwind, then the opposite of what "should" happen, might occur. That is a human error.From Battle of the Quants - a New York seminar.

At the below extract....WOW

Loved TJ's post below - bang on the money (though I have a strong vested interest). Couple of questions:"And once the programming and testing is done you have eliminated key-person risk."...surely only until the key-person exits with the IP? "...trade crowding problems." Bit (a lot) of a tangent to this, but would be very interested for any views on "time frame crowding". We seem to have seen a huge rush to the microstructure level to capture return and the "tech arb" of ever faster networks/machines/code etc seems to be approaching the buffers of the possible. Does anyone have a view on the possible reversion to longer time frames to escape the battle over an extremely narrow and decreasing strip of turf?

QuoteOriginally posted by: PaulTo add to g000RRRe's point, you'd be surprised (horrified perhaps, like me) by the amount of trading that goes on using ideas that haven't been tested or even ideas that have been tested and shown to not work.PI can understand that somebody invests the money in a bad strategy if it is well designed, but based on the erroneous premises (e.g. LTCM). Such strategies "make sense" and are usually backtested. The problems arise because of overconfidence and fragile risk exposure (the #2 traders' worst enemy: greed).But, why do people invest the money in algoritmic trading strategies that are not even tested or don't work? This doesn't make sense. Unless it's the fraud or Ponzi scheme.

QuoteOriginally posted by: Traden4AlphaThis view assumes 1) that a trading strategy is a passive extractor of exogenously-supplied excess returns -- i.e., that the trader merely harvests preexisting inefficiencies; 2) that all traders are equivalent (deterministically or stochastically) in their ability to recognize patterns and execute trades against them; 3) that traders (or their employers) are rational. None of the assumptions is right and that creates the potential for persistent profits for some traders (and persistent losses for others).I didn't have in mind regular, human trading, but algoritmic trading. There is a big difference.1) Algoritmic trading indeed is a "passive extractor of exogenously-supplied excess returns -- i.e., that the trader merely harvests preexisting inefficiencies". 2) My assumption was that most research departments utilize same historical data and use similar data mining techniques. Since there are so many of them, market efficiency should indeed work, at least with computer trading, to some extent. Human traders are naturally much more heterogenous.3) Traders are irrational, but the computers, data mining techniques and researchers are very rational, and what's worse, very homogenous.

QuoteConsider an observable arbitrage opportunity and the dynamics by which traders discover the pattern and trade it. The first trader to find the pattern will have the pattern all to themselves and will profit for a time. But what happens when a second (or additional) trader finds the pattern? Will they equally share in the profits with incumbent trader(s)? The answer is no. Traders with faster decision-making and execution platforms will take more of the pattern's profit than traders with slower execution platforms. What's interesting is that traders with lagging execution may actually help boost the profitability of traders with leading execution. One can even imagine 110%:-10% split of the profits -- slow traders may bleed capital to fast traders. Moreover, for some broad categories of trading patterns (including momentum trading) the growth in the amount of capital applied to the pattern strengthens the pattern -- i.e. the more traders that trade on momentum, the more momentum one will see. The result is a persistent pattern that attracts new traders who, if they have a slower decision-making and execution, will bleed capital to faster traders. (Traders will also systematically assort themselves on transactional overhead and risk-tolerance in front-running any pattern.)But, this is very good point. Slow momentum traders can serve as a great dump for unloading the early buyers, unless there is a strong fundamental reason for the momentum. In that case, long term opposite side pays the price.I would add that for some other trading strategies (e.g. breakout patterns), the cycle would develop like in every other business. Early developer would earn extra profits at the early stage. Subsequently, other competitors jump in, causing profit margins to narrow (e.g. more false breakouts in the stock markets, compared with the markets 50 years ago). At the end, the strategy will survive, but with fewer players and sustainable profits.

QuoteNow, if you assume that pure logicals rules don't work, then its a whole different discussion of wheter intuition exists and chartists really do have something that cannot be translated into codes or if it is just rational rules with bits of randomness blurring the image.If markets are rational, efficient and random, no trading strategy would work, either algoritmic or human. Human behavioral patterns that are irrational do exibit some predictability that can be rationally explained (fear, greed, manipulation) and thus exploited. Intuition should be superior to rational explanation. Intuition is the tool of the subconscious mind that doesn't need rationalization, but directly processes the information, like a "creative reflex" (oxymoron, I know). But, intuitive trading should only be working if there are no destructive beliefs blocking the reflex. Also, it can only be the result of the years of experience staring at the price movement since it is the only way to develop proper reflex. Mark Douglas in "Trading in the zone" even uses analogy with Jung's "collective unconscious". According to him, traders enter the colloective unconsciousness of the market and that's when intuition comes to play. It is also interesting idea.I also think that there is a room for intuition when there is information overload and intoxication, so rational mind cannot process all of it. That's what happens if you read WSJ or Bloomberg too much. Some get a headache, others use intuition. But, all this is just a theory. I have never heard of a real human trader that trades solely on intuition. If such traders exist, they are probably silent because they are afraid of being attached "wierdo" attribute. What do you think: are the legends true?