Hi Stephen,I have to disagree with this statement. When you're dealing with real markets, the problem is that "all available optimization criteria" are not knowable, and they change over time. Control theory is wonderful when all of the parameters are known - in markets these shift all the time (for example, over a week the NASDAQ may steadily outperform, but then get hammered the next day as MSFT preannounces).Furthermore in a high frequency trading situation the percent of trades that are profitable and expected return across all trades will tend to be low. Percentage of trades that are profitable will be less than 60%, unless you are doing a speed-based pure arbitrage (which is not what you're describing). Expected return across all trades will tend to be low (again excluding speed-based pure arbitrage), as highly profitable patterns are quickly discovered and exploited away. Basically high frequency arbitrage creates steady profits by taking low positive expectation per trade and multiplying by many trades.Please keep in mind what I'm describing below comes not from theory, but from plenty of experience trading a variety of different kinds of positions.In any individual trade, you will have several classes of outcomes: (1. MAX LOSS) your maximum loss threshhold will be reached, forcing you to blow out the trade (stop loss), (2. TAKE LOSS) you will exit at some acceptable loss when it is advantageous to do so, (3. PROFIT) you will exit at a profit.Consider an example where you trade SPY, up to 2000 shares. You plan to put the position on in pieces (say 200 shares each) to minimize market impact and optimize your entry.You should find that most of the time when you hit MAX LOSS, you have most of the 2000 shares in place. Or, perhaps you have 1000 shares in place and the move was twice what you would have taken with 2000 shares in place. Regardless, on the worst trades, you will find that you were actually able to put on the position fairly easily.In the PROFIT case, you will generally find it was much harder to get the entire position in place. When the trade is a winner, the market is often moving in your favor, there are a lot of other people trying to do the same trade, and it gets hard to get into the position that will eventually be a winner. Maybe you can only get 800 of the 2000 shares, or maybe only 1200, or 200.All of the math and theory aside, this is a simple reality of trading: It's much harder to get into a winning position than a losing one.You see the problem that starts to develop if all of your MAX LOSS trades involve large positions, and your PROFIT trades cannot get entire positions. The expected profit based on your research essentially goes to hell. In high frequency arbitrage, you must make sure that you get as much of the position in place as possible when the trade is going to be in the PROFIT category. Speed becomes an essential part of getting that done.There are lots of other things I could say on why speed matters in high frequency arbitrage.I suppose the most important is that the 2-3 shops that I have encountered that do high-frequency arbitrage with great success all emphasize speed. I know one of these houses puts a high degree of emphasis on marketable FOK orders.My guess is that you will be able to rationalize away why speed doesn't matter while you do your research, but when you go live you will realize it is quite important.With all due respect to farmer, I have looked at his website, and I don't think he does high frequency arbitrage or has ever done high frequency arbitrage. When you haven't done it, it's easy to theorize about reality what might be.Ultimately, the debate here can only be settled one way. You would have to show me a high-frequency arbitrage strategy which over a reasonable period of time in live trading does well with suboptimal speeds and where performance improvement with optimal speeds would be marginal at best.Until then, my experience tells me that in high frequency arbitrage, speed matters. Theorizing on other possibilities will not change that, so I suppose may stop posting on this aspect of the discussion now.Best Regards,MJQuoteOriginally posted by: crowlogicSplitting your order into optimal, tiny chunks based on all available optimization criteria (be creative) will *ALWAYS* be more effective than ad-hoc rule based logic.You should look into control theory...

QuoteOriginally posted by: genie92In any individual trade, you will have several classes of outcomes: (1. MAX LOSS) your maximum loss threshhold will be reached, forcing you to blow out the trade (stop loss), (2. TAKE LOSS) you will exit at some acceptable loss when it is advantageous to do so, (3. PROFIT) you will exit at a profit.Consider an example where you trade SPY, up to 2000 shares. You plan to put the position on in pieces (say 200 shares each) to minimize market impact and optimize your entry.You should find that most of the time when you hit MAX LOSS, you have most of the 2000 shares in place. Or, perhaps you have 1000 shares in place and the move was twice what you would have taken with 2000 shares in place. Regardless, on the worst trades, you will find that you were actually able to put on the position fairly easily.In the PROFIT case, you will generally find it was much harder to get the entire position in place. When the trade is a winner, the market is often moving in your favor, there are a lot of other people trying to do the same trade, and it gets hard to get into the position that will eventually be a winner. Maybe you can only get 800 of the 2000 shares, or maybe only 1200, or 200.All of the math and theory aside, this is a simple reality of trading: It's much harder to get into a winning position than a losing one.This is all crank material. Why is your entry price an input into a trading strategy? It only has any predictive use to the extent you are acting on it. And even then, only to a person who wishes to run your stop. Of course you should be replicating his strategy, given you know where your own stop is. But if you did that, you would realize the pointlessness. If your entry price is simply a good indicator of recent prices, or of what other people are doing somehow, you should use recent prices or try to get that other variable directly, not use your entry price as an estimate.Let's suppose that your plan is to exit all positions when your strategy is flailing, or has an unusually bad run of predictions. And let's suppose that you decide to be lazy, and use the profit or loss on your immediate "individual trade" to measure this. Even so, this plan to exit all positions under circumstance X is still an element of your strategy. Therefore, to follow it, is to assume your predictive ability is still valid. If you assume that the correlation of your predictions to the actual market will be random, then you should reduce your risk exposure right away, but gradually, or "when it is acceptable to do so" assuming you think can still make that call. But if you think your strategy has what I might call "tragically negative" correlation to the market - such that even a decision to execute the opposite of your strategy would be made at the wrong time - then you should use a random-number generator to time your exit, and probably without even figuring your cost of random risk exposure into the distribution of exit times.I also can only laugh at the statement "it gets hard to get into the position that will eventually be a winner." Sure, it is much easier to buy a Ferrari for a billion dollars than for a hundred. By definition, if you enter a delayed series of orders at one price, the condition of the second one getting filled rules out the condition of the first one showing a profit at the time the second one is filled, but does not rule out the condition of the first one showing a loss at the time the second one gets filled. This does not, however, prove that you would have liked to buy at all at once. The condition of the second one getting filled means you would have lost just as much if you filled them both at the price and time when you filled the first one. But the first one showing a profit at some time before the second one is filled, does not prove it was correct to attempt to buy an unlimited amount at the time you actually purchased a finited amount.The reason someone enters an order, is because he predicts prices will be higher. You are saying that if you are predicting prices will rise, there is no reason you should postpone any part of your buy order. But consider that your own knowledge that you are going to sell, gives you a prediction that prices will fall in the future, or rise less than they might have, at the moment you sell. Put simply, you buy until you no longer predict the market will rise, at which point you believe the market will be flat, when compared to the acceptable or market rate of return. And any additional buys would have a negative "real" expectation.But suppose after you have bought ths ideal position, some new information then causes you to think the market will rise even more, so you buy some more. The position you now hold is a single position - it only takes one function call to calculate its risk in a computer program - but would only be classfied as "an individual trade" by a crank.Just because you predict the market would rise if you got struck by lightening, does not make it impossible for you to buy at a price where you would lose money. Suppose the Fed buys bonds once a day, and when they do, people sell the bonds and use the money to bid on Coke stock at 20 times earnings. You might predict that the price of Coke stock will go up 1 cent over a year, and that you will be able to sell a million shares at the new price. But you would still want to spread your million buys out. Even in super-short-term trading, you can buy too fast simply by buying faster than anybody with a limit order predicted you would, or is used to. People place limit orders because they know there is a good chance you will come sweeping in. For this to work, you have to fulfill their expectations, by sweeping at an orderly rate.Your economics is so nonsensical, I won't bother with any more of it. The question is, are you going to take the time to actually get good at it? Or are you going to quit after less time than you spent, for example, learning to program?

QuoteOriginally posted by: genie92Hi Stephen,I have to disagree with this statement. When you're dealing with real markets, the problem is that "all available optimization criteria" are not knowable, and they change over time. Control theory is wonderful when all of the parameters are known - in markets these shift all the time (for example, over a week the NASDAQ may steadily outperform, but then get hammered the next day as MSFT preannounces).With all due respect, I think you completely missed my point. Have you studied control theory? If so, you would know that in real situations that engineers encounter on a daily basis, the complete set of inputs and relationships is seldom ever known, if it is then it is a linear system and is completely controllable.You have 1 optimization criteria, to make money. This criteria can't be thrown into a closed formula though. So you break it down some more, the criteria is to minimize drawdown, maximize P/L and minimize commissions/fees. A direct access broker is gonna pass the exchange fees and rebates directly to you, so every exchange has its own pricing and this can vary between order types, liquidity supplier vs consumer, etc. Order routing must be optimized. Expected order round-trip times must be optimized as well.I have no idea why the nasdaq outperforming would have any bearing on a high frequency system, this would imply that you are taking long-only positions and are in the market a significant enough amount of time to be carried up with it.QuoteFurthermore in a high frequency trading situation the percent of trades that are profitable and expected return across all trades will tend to be low. Percentage of trades that are profitable will be less than 60%, unless you are doing a speed-based pure arbitrage (which is not what you're describing). Expected return across all trades will tend to be low (again excluding speed-based pure arbitrage), as highly profitable patterns are quickly discovered and exploited away. Basically high frequency arbitrage creates steady profits by taking low positive expectation per trade and multiplying by many trades.Please keep in mind what I'm describing below comes not from theory, but from plenty of experience trading a variety of different kinds of positions.Yes, of course every trade is going to have a low expectency, that is all the more reason to use a more rigorous method.QuoteIn any individual trade, you will have several classes of outcomes: (1. MAX LOSS) your maximum loss threshhold will be reached, forcing you to blow out the trade (stop loss), (2. TAKE LOSS) you will exit at some acceptable loss when it is advantageous to do so, (3. PROFIT) you will exit at a profit.Consider an example where you trade SPY, up to 2000 shares. You plan to put the position on in pieces (say 200 shares each) to minimize market impact and optimize your entry.You should find that most of the time when you hit MAX LOSS, you have most of the 2000 shares in place. Or, perhaps you have 1000 shares in place and the move was twice what you would have taken with 2000 shares in place. Regardless, on the worst trades, you will find that you were actually able to put on the position fairly easily.In the PROFIT case, you will generally find it was much harder to get the entire position in place. When the trade is a winner, the market is often moving in your favor, there are a lot of other people trying to do the same trade, and it gets hard to get into the position that will eventually be a winner. Maybe you can only get 800 of the 2000 shares, or maybe only 1200, or 200.I have no idea why you would want to decide upon the number of 2000 shares at the outset and then implement some arbitrary decision to split it into chunks of 100 over time. You are making an awful lot of assumptions, and you probably came up with these numbers cause they are nice and round and odd-lots can be more expensive.I optimally shift cash to the highest expectency strategies and use the exchange fees and the conditional expectations of evertything in my universe to decide when to trade, I don't have any hard-coded logic or some fixed percentage of buying power to decide positions. QuoteAll of the math and theory aside, this is a simple reality of trading: It's much harder to get into a winning position than a losing one.You see the problem that starts to develop if all of your MAX LOSS trades involve large positions, and your PROFIT trades cannot get entire positions. The expected profit based on your research essentially goes to hell. In high frequency arbitrage, you must make sure that you get as much of the position in place as possible when the trade is going to be in the PROFIT category. Speed becomes an essential part of getting that done.There are lots of other things I could say on why speed matters in high frequency arbitrage.I suppose the most important is that the 2-3 shops that I have encountered that do high-frequency arbitrage with great success all emphasize speed. I know one of these houses puts a high degree of emphasis on marketable FOK orders.My guess is that you will be able to rationalize away why speed doesn't matter while you do your research, but when you go live you will realize it is quite important.QuoteI am not saying speed doesn't matter at all, but speed alone isnt going to make you any money, and I have traded in the past but was disgusted with all of the available tools and platforms and realized emotions should not be involved in decision making, AT ALL. Also, a human can never beat a specfially crafted and well-tuned machine.My system receives quotes from all major exchanges at a colo in austin Regular ping times are 40ms to my broker, and the latency on the quotes from the exchanges is around 45 to 50ms.. when I move the box to Lime's colo the ping times to the internal servers is 1ms and from there to the exchange is another 5 or 6. Is that fast enough for you? So colocated my quote-lag-time is about 5ms and my order-round-trip is about 10ms. Yes, filled that quick.. Lime does *huge* volume. Also, this API is in java and only costs $200 a month for colo fees..And it took me nearly 0 time and effort to get this kind of performance, and you think it's gonna help me by spending $30,000 to get a few feet closer to the exchange and rewrite the whoel thing in hand-tuned assembler?I'm not saying speed isnt important, I am saying it is absurd to spend a lot of time and money optimizing this instead of high level strategy design.With all due respect to farmer, I have looked at his website, and I don't think he does high frequency arbitrage or has ever done high frequency arbitrage. When you haven't done it, it's easy to theorize about reality what might be.Ultimately, the debate here can only be settled one way. You would have to show me a high-frequency arbitrage strategy which over a reasonable period of time in live trading does well with suboptimal speeds and where performance improvement with optimal speeds would be marginal at best.Until then, my experience tells me that in high frequency arbitrage, speed matters. Theorizing on other possibilities will not change that, so I suppose may stop posting on this aspect of the discussion now.Definately.. the proof is in the eating. Bear in mind that I'm not just making this stuff up from theories in my head, this is all based on empirical observations of the 2-3GB of data I record each day, and simulated orders over a range of lag-time-simulations... using full depth of book data from all major equity exchanges.If I was in a rush I could throw whatever I have now together and start trading but I am not satisified battling for pennies at the sub-second level.Good luck,Stephen

QuoteOriginally posted by: genie92With all due respect to farmer, I have looked at his website, and I don't think he does high frequency arbitrage or has ever done high frequency arbitrage. When you haven't done it, it's easy to theorize about reality what might be.Oops, I didn't read that far. But you don't really know what you are talking about, genie92. It took you like 10 posts to even clear up the fact that you didn't mean what you call "pure arbitrage," just really high-frequency trading, i.e. scalping. And who among us hasn't scalped? Since both me and crowlogic have obviously thought about it, you can guess that either A) we actually tried it for longer than you before quitting like you, or B) we came to the same conclusion as you faster than you.I was a SOES bandit in the office of Harvey Houtkin in 1993, when SOES was the only electronic execution system in the world with any volume. I sat up all night watching currency bids and offers move on Globex when the total volume for the night might be zero. I worked for the highest-volume day-trader of the Nasdaq boom. I succesfully predicted the trading patterns and volume characteristics of OptiMark, Liquidnet, Primex, and the NYSE going to pennies, before they came on line, and going against the predictions of the people who designed them. I proposed what the CBOT calls their "LDB" data feed four years before they did. I bought the low tick and sold the high tick in the e-mini S&P two days in a row in 1998.I may not know any math, but I watched this high-frequency world grow up from nothing. I watched the speeds get faster and faster, without the game really changing. But what's to theorize about? You said it yourself,QuoteOriginally posted by: genie92As far as trading strategies, some of the strategies are very easy if you're fast enough. Unfortunately it takes a lot of monitoring, a lot of time, and paying a lot of fees to get everything working and profitable in real-time. Not to mention- there's a whole heck of a lot of risk of things going wrong!

Quotecrowlogic: If I was in a rush I could throw whatever I have now together and start trading but I am not satisified battling for pennies at the sub-second level.Why not? RenTec have made billions doing this.

Quotefarmer: I may not know any mathAnd you want people to invest how much in your "hedge fund"? LOL.

I'm trying to emulate Buffet. He doesn't know any math either.QuoteOriginally posted by: TraderJoeQuoteI'm saying that trying to emulate Buffet isn't going to get you shit.I disagree entirely.So should I learn math? Or do I not actually need to?QuoteOriginally posted by: TraderJoeAsk Buffett -and he has the proof to the tune of $40billion.

QuoteOriginally posted by: farmerI'm trying to emulate Buffet. He doesn't know any math either.QuoteOriginally posted by: TraderJoeQuoteI'm saying that trying to emulate Buffet isn't going to get you shit.I disagree entirely.So should I learn math? Or do I not actually need to?QuoteOriginally posted by: TraderJoeAsk Buffett -and he has the proof to the tune of $40billion.Different strokes for different folks. Hey, if you ever change your mind...QF Math.

To crowlogic: I will be curious to hear how everything goes when you finally turn live. As you said, the proof will be in the eating . I think our debate and communication has been useful, and I do wish you the best in your venture. Please update me on how you progress.To farmer: It seems from your descriptions of getting beat up trading oil on your website that you are trading futures. I'm curious to hear after all your "experience" whether you even have the required CTA or CPO designation to trade futures for outside accounts. I checked the NFA database, but I must be looking under the wrong names, or perhaps you're not using futures?It's also interesting that you state on your site that your neural-net trading strategy is your "most robust" strategy. Then you say you just created the program at the start of 2006 with only theoretical results to 1999. Like most of your other comments, that doesn't make sense to me.I'm not finding your comments to have value, so I think it's best to end our discussions.

I think you just answered your question. RenTec is already doing this and can afford the hardware needed to squeeze out another ms here and there.I already know guys relying on this type of strategy exclusively and they are constantly shitting their pants worrying if the network is gonna hiccup for 1-second, or some other weird thing like that.QuoteOriginally posted by: TraderJoeQuotecrowlogic: If I was in a rush I could throw whatever I have now together and start trading but I am not satisified battling for pennies at the sub-second level.Why not? RenTec have made billions doing this.

QuoteOriginally posted by: genie92I'm not finding your comments to have value, so I think it's best to end our discussions.Well you just keep me updated as to when you "suppose may stop posting on this aspect of the discussion now." I'll put it in my calendar and build my day around it. Did you decide I was not worth wasting time talking to before or after wasting your time searching the regulatory databases for my name? That is after all what the databases are there for, to help people who are unqualified to actually talk finance in screening investment professionals. So you did exactly what someone with $500 to put in a mutual fund should do in that situation.QuoteOriginally posted by: genie92Then you say you just created the program at the start of 2006 with only theoretical results to 1999. Like most of your other comments, that doesn't make sense to me.Well obviously not, or you could give me some constructive criticism. But given that you have almost zero knowledge or experience trading, you have nothing to say. I stand by the statement on my web site. I have no more robust strategy, or strategy which I would have more confidence recommending to someone with a conservative diversification objective. In fact, I know of know money manager with any track record over any number of years, who has a strategy which I would judge to be more robust than the one under discussion. But then I have tools to evaluate robustness, as well as to define it, and you do not. I am qualified to make such a judgement, and you are not. In fact, you are not even qualified to have such a judgement made on your behalf.

QuoteOriginally posted by: crowlogicQuoteOriginally posted by: MikeCroweQuoteOriginally posted by: crowlogicDon't get me wrong, I think OOP is a wonderful concept, I just don't believe it is the answer to life the universe and programming. The part I object to is writing static (or shared in VB) classes/methods. OOP tells us that we should think in terms of objects and then build the functionality of that object in with it. Creating a class that you never create an object of yet can still access its methods is completely against this, but is required because not everything you want to do is an object. I much prefer to say that if something is described well by an object, then use OOP, if not then use functional programming. When it comes to compile time all it does is compile it as though I'd written static classes but it makes my code much more logical. As far as I can see static methods is functional programming in OOP clothing.Well, it's certainly not the answer to everything, complexity and biologically-inspired neural simulation probably is getting closer, but we have a ways to go yet. You are absolute correct with regard to static methods being used as functional programming, it's not OOP at all.I avoid static methods like the plague as well.. a good design will not require any static methods in ideal situations.Can you name some instances where static/functional methods are absolutely required rather than designing your classes in some other way?Maybe not "absolutely required" but then OOP isn't absolutely required either . I prefer to use a simpler more logical route rather than forcing it into a overly complex class structure. A simple example springs to mind that various things I've done need finding the greatest common divisor of a series of numbers, or lowest common multiple, I find this simplest to implement functionally since there isn't an associated object and many different programs/classes use this. Also standard functions such as beta, gamma, normal etc. There could be a case for doing random number generators in a class, but I've never bothered. Basically anything that I'll use in multiple projects/classes that are not conceptually connected I build as a function - this is almost always just some maths routine or formula.I am concious that this is now v. off thread so I'll drop it.

Can you please explain what you know about neural-nets farmer to give me some confidence to give you $10million. Many thanks.

QuoteOriginally posted by: TraderJoeCan you please explain what you know about neural-nets farmer to give me some confidence to give you $10million. Many thanks.Yes, without giving away the farm, farmer, can you tell me general types of neural nets you use?There are hundreds of different mathematical techniques which fall under the broad class of neural.. most of which are complete crap, and some of which, are closer to the answer to everything. k-NN, feed forward, recurrent, time-delay, dynamic, first order, second urder training, unscented, filtering, generatic topograhic mappings, auto-associative, long-term-short-term, hierarchal, etc?

QuoteOriginally posted by: crowlogicQuoteOriginally posted by: TraderJoeCan you please explain what you know about neural-nets farmer to give me some confidence to give you $10million. Many thanks.Yes, without giving away the farm, farmer, can you tell me general types of neural nets you use?There are hundreds of different mathematical techniques which fall under the broad class of neural.. most of which are complete crap, and some of which, are closer to the answer to everything. k-NN, feed forward, recurrent, time-delay, dynamic, first order, second urder training, unscented, filtering, generatic topograhic mappings, auto-associative, long-term-short-term, hierarchal, etc?As you've just given away the answer, I'll have to think of another question now.