Serving the Quantitative Finance Community

 
User avatar
maxw
Topic Author
Posts: 0
Joined: December 5th, 2012, 11:16 pm

HFT Algorithms

December 4th, 2013, 2:16 am

Hello All,I'm new to hft (have collocated box in NYC). I just got started in August with little success. I had been researching and implemented about 10 strategies which relied on pattern recognition, machine learning, beysian algorithms, random motion prediction as well as some artificial intelligence. All failed miserably. I did try triangular arbitrage recently as I'm mainly in forex. The problem here is execution. Sometimes you get stuck one position in when you have two more to go and then you have to back out while making a loss. Price correction happens very quickly even orders within 30 milliseconds won't be of any help sometimes. So now I want to start from a more informed starting point. What are algorithms hft folk use ... in general terms or specific if you care to share.Sincerely,MAXW
 
User avatar
jige
Posts: 0
Joined: December 4th, 2011, 4:08 pm

HFT Algorithms

December 4th, 2013, 4:46 pm

Hello,I wont get into specifics HFT algos because I dont know but Ill just take a quick moment to focus on your 30 milliseconds.I'm not from a HFT background, I'm mainly focused on options.1. Increasing speed wont change anything if there is an issue on how you designed or exec your models, so if you can increase speed easily in your architecture do it, otherwise maybe check other things first.2. you should sent all your orders at the same time, so the window shall not be 30 millisec, its way too much to do HFT:if i ping the wilmott server i got an answer within 25ms, meaning in 25ms I send the request, the server receive it, send me back the answer and the answer arrives back at me.This is just using windows basic protocol. So you shall be able to do better than 30ms.Once you detect an opportunity you send all your orders at the SAME time (so probably in the same couple of microsec or tens of nano ?)3 To compare: even on Options i can get time around 50 micro so if the underlying move quickly on high percentage delta I can Lift/Hit some slower algos, it happens quite often4. Improve your speed by doing less checks before sending the orders or use a different language or try to parallelize more things...After im sure that some HFT guys have not much better than 30ms and still making money on their algos.
Last edited by jige on December 3rd, 2013, 11:00 pm, edited 1 time in total.
 
User avatar
AnalyticalVega
Posts: 104
Joined: January 16th, 2013, 5:03 am

HFT Algorithms

December 5th, 2013, 10:12 am

QuoteOriginally posted by: maxwHello All,I'm new to hft (have collocated box in NYC). I just got started in August with little success. I had been researching and implemented about 10 strategies which relied on pattern recognition, machine learning, beysian algorithms, random motion prediction as well as some artificial intelligence. All failed miserably. I did try triangular arbitrage recently as I'm mainly in forex. The problem here is execution. Sometimes you get stuck one position in when you have two more to go and then you have to back out while making a loss. Price correction happens very quickly even orders within 30 milliseconds won't be of any help sometimes. So now I want to start from a more informed starting point. What are algorithms hft folk use ... in general terms or specific if you care to share.Sincerely,MAXWSounds like you may be doing many things wrong.1) What order types are you using?2) are you posting bids when attempting to buy?3) What is your cancellation rate?4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?5) Are you using algorithm switching in realtime? Are you paper trading your algorithms in parallel and switching to the most profitable one based on well defined statistical metrics?6) How well do you understand market microstructure?7) Are you backtesting your strategies with solid mathematical statistics? Forward Testing with Monte Carlo? I'm certainty not an expert in HFT, but you need to have good answers to the above questions to have any chance I would think.
 
User avatar
chocolatemoney
Posts: 0
Joined: October 8th, 2008, 6:50 am

HFT Algorithms

December 6th, 2013, 6:58 am

QuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: maxwHello All,I'm new to hft (have collocated box in NYC). I just got started in August with little success. I had been researching and implemented about 10 strategies which relied on pattern recognition, machine learning, beysian algorithms, random motion prediction as well as some artificial intelligence. All failed miserably. I did try triangular arbitrage recently as I'm mainly in forex. The problem here is execution. Sometimes you get stuck one position in when you have two more to go and then you have to back out while making a loss. Price correction happens very quickly even orders within 30 milliseconds won't be of any help sometimes. So now I want to start from a more informed starting point. What are algorithms hft folk use ... in general terms or specific if you care to share.Sincerely,MAXWSounds like you may be doing many things wrong.1) What order types are you using?2) are you posting bids when attempting to buy?3) What is your cancellation rate?4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?5) Are you using algorithm switching in realtime? Are you paper trading your algorithms in parallel and switching to the most profitable one based on well defined statistical metrics?6) How well do you understand market microstructure?7) Are you backtesting your strategies with solid mathematical statistics? Forward Testing with Monte Carlo? I'm certainty not an expert in HFT, but you need to have good answers to the above questions to have any chance I would think.4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?What do you mean by PDF? probability density function ? Why would you need that?
 
User avatar
farmer
Posts: 61
Joined: December 16th, 2002, 7:09 am

HFT Algorithms

December 6th, 2013, 10:41 am

QuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.
 
User avatar
chocolatemoney
Posts: 0
Joined: October 8th, 2008, 6:50 am

HFT Algorithms

December 6th, 2013, 11:13 am

QuoteOriginally posted by: farmerQuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.I have never seen anyone "forward testing" a strategy "with Monte Carlo". What the heck does it add to your trading system design?
 
User avatar
AnalyticalVega
Posts: 104
Joined: January 16th, 2013, 5:03 am

HFT Algorithms

December 6th, 2013, 2:40 pm

QuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: maxwHello All,I'm new to hft (have collocated box in NYC). I just got started in August with little success. I had been researching and implemented about 10 strategies which relied on pattern recognition, machine learning, beysian algorithms, random motion prediction as well as some artificial intelligence. All failed miserably. I did try triangular arbitrage recently as I'm mainly in forex. The problem here is execution. Sometimes you get stuck one position in when you have two more to go and then you have to back out while making a loss. Price correction happens very quickly even orders within 30 milliseconds won't be of any help sometimes. So now I want to start from a more informed starting point. What are algorithms hft folk use ... in general terms or specific if you care to share.Sincerely,MAXWSounds like you may be doing many things wrong.1) What order types are you using?2) are you posting bids when attempting to buy?3) What is your cancellation rate?4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?5) Are you using algorithm switching in realtime? Are you paper trading your algorithms in parallel and switching to the most profitable one based on well defined statistical metrics?6) How well do you understand market microstructure?7) Are you backtesting your strategies with solid mathematical statistics? Forward Testing with Monte Carlo? I'm certainty not an expert in HFT, but you need to have good answers to the above questions to have any chance I would think.4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?What do you mean by PDF? probability density function ? Why would you need that?Yes. You would need to reconstruct the most likely prices in realtime, so your sample space for your empirical PDF would constantly be changing. The rate of change of the sample space would depend on market conditions which would need to be measured by a bunch of different metrics.
 
User avatar
AnalyticalVega
Posts: 104
Joined: January 16th, 2013, 5:03 am

HFT Algorithms

December 6th, 2013, 2:42 pm

QuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: farmerQuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.I have never seen anyone "forward testing" a strategy "with Monte Carlo". What the heck does it add to your trading system design?Are you serious? How do you predict your future P&L?
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

HFT Algorithms

December 6th, 2013, 3:33 pm

QuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: farmerQuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.I have never seen anyone "forward testing" a strategy "with Monte Carlo". What the heck does it add to your trading system design?Are you serious? How do you predict your future P&L?Indeed! And how do you know if your strategy development process doesn't have a leak (accidentally using future data to trade the past) or has overfitted the backtest data?
 
User avatar
chocolatemoney
Posts: 0
Joined: October 8th, 2008, 6:50 am

HFT Algorithms

December 6th, 2013, 4:21 pm

QuoteOriginally posted by: Traden4AlphaQuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: farmerQuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.I have never seen anyone "forward testing" a strategy "with Monte Carlo". What the heck does it add to your trading system design?Are you serious? How do you predict your future P&L?Indeed! And how do you know if your strategy development process doesn't have a leak (accidentally using future data to trade the past) or has overfitted the backtest data?@AnalyticalVeg@Traden4Alpha: Literature is rich on the topic, isn't it?
 
User avatar
AnalyticalVega
Posts: 104
Joined: January 16th, 2013, 5:03 am

HFT Algorithms

December 6th, 2013, 5:12 pm

QuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: Traden4AlphaQuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: farmerQuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.I have never seen anyone "forward testing" a strategy "with Monte Carlo". What the heck does it add to your trading system design?Are you serious? How do you predict your future P&L?Indeed! And how do you know if your strategy development process doesn't have a leak (accidentally using future data to trade the past) or has overfitted the backtest data?@AnalyticalVeg@Traden4Alpha: Literature is rich on the topic, isn't it?Many people use Monte Carlo. Do you have something that works better?
Last edited by AnalyticalVega on December 5th, 2013, 11:00 pm, edited 1 time in total.
 
User avatar
maxw
Topic Author
Posts: 0
Joined: December 5th, 2012, 11:16 pm

HFT Algorithms

December 9th, 2013, 11:54 am

1) What order types are you using?I use limit orders and only use market orders when I'm trying to unwind a position that has gone bad on me and I want to get out. 2) are you posting bids when attempting to buy?No I don't do that. I buy at the ask, or sell at the bid so I get quick fills.3) What is your cancellation rate?I'd say 60%.4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?5) Are you using algorithm switching in realtime? Are you paper trading your algorithms in parallel and switching to the most profitable one based on well defined statistical metrics?No I'm not yet at this point. The algorithms I did have didn't really work too well. They weren't really geared towards financial trading, but to general prediction and pattern recognition. I used libraries usually used for research in other fields. 6) How well do you understand market microstructure?I've been studying it for 5 years on a part time basis. But there is still so much to learn.7) Are you backtesting your strategies with solid mathematical statistics? Forward Testing with Monte Carlo? I did do back testing, but I overlooked on small but very very important factor - getting filled. I thought I'd always get filled. It doesn't work like that (you have to wait in line with everybody else unless you are buying at the ask or selling at the bid).
 
User avatar
maxw
Topic Author
Posts: 0
Joined: December 5th, 2012, 11:16 pm

HFT Algorithms

December 9th, 2013, 11:56 am

Indeed! And how do you know if your strategy development process doesn't have a leak (accidentally using future data to trade the past) or has overfitted the backtest data? ----This is the problem I ran into. But I'll give the Monte carlo algorithm a try and see what happens.
 
User avatar
AnalyticalVega
Posts: 104
Joined: January 16th, 2013, 5:03 am

HFT Algorithms

December 9th, 2013, 1:22 pm

QuoteOriginally posted by: maxw1) What order types are you using?I use limit orders and only use market orders when I'm trying to unwind a position that has gone bad on me and I want to get out. 2) are you posting bids when attempting to buy?No I don't do that. I buy at the ask, or sell at the bid so I get quick fills.3) What is your cancellation rate?I'd say 60%.4) Are you reconstructing PDFs of basic market metrics in realtime? How do you compute the current market state? How many market states do you have in your models?5) Are you using algorithm switching in realtime? Are you paper trading your algorithms in parallel and switching to the most profitable one based on well defined statistical metrics?No I'm not yet at this point. The algorithms I did have didn't really work too well. They weren't really geared towards financial trading, but to general prediction and pattern recognition. I used libraries usually used for research in other fields. 6) How well do you understand market microstructure?I've been studying it for 5 years on a part time basis. But there is still so much to learn.7) Are you backtesting your strategies with solid mathematical statistics? Forward Testing with Monte Carlo? I did do back testing, but I overlooked on small but very very important factor - getting filled. I thought I'd always get filled. It doesn't work like that (you have to wait in line with everybody else unless you are buying at the ask or selling at the bid).You are doing a lot things incorrectly. The people who make money in HFT do things much differently. Their thought process is different as well.This forum is not a good place to get you up to speed. If I were you, I would look for a good mentor.
Last edited by AnalyticalVega on December 8th, 2013, 11:00 pm, edited 1 time in total.
 
User avatar
chocolatemoney
Posts: 0
Joined: October 8th, 2008, 6:50 am

HFT Algorithms

December 9th, 2013, 2:34 pm

QuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: Traden4AlphaQuoteOriginally posted by: AnalyticalVegaQuoteOriginally posted by: chocolatemoneyQuoteOriginally posted by: farmerQuoteOriginally posted by: AnalyticalVegaForward Testing with Monte Carlo?Easy there, fella. Find someone your own size to pick on. The people who run Monte Carlo need to eat, also.I have never seen anyone "forward testing" a strategy "with Monte Carlo". What the heck does it add to your trading system design?Are you serious? How do you predict your future P&L?Indeed! And how do you know if your strategy development process doesn't have a leak (accidentally using future data to trade the past) or has overfitted the backtest data?@AnalyticalVeg@Traden4Alpha: Literature is rich on the topic, isn't it?Many people use Monte Carlo. Do you have something that works better?If Monte Carlo suits your need, sure, go ahead.My point is: pay attention! You have still to make an assumption on the process and that same assumption may be already embedded in your model. Therefore, forward testing may not help you stretch your assumption and identify those edge cases where things may get very ugly. In other words, over the years, I spent significant time improving on the schoolbook forward testing Monte Carlo, building a richer framework. My all point is not about Monte Carlo, but schoolbook implementation of forward testing with Monte Carlo.Second thing. I see that you have your checklist, which is good. However, you can do all you say and still lose money and maybe do something completely different and bring home your profit.On the same line: You say "Yes. You would need to reconstruct the most likely prices in realtime, so your sample space for your empirical PDF would constantly be changing. The rate of change of the sample space would depend on market conditions which would need to be measured by a bunch of different metrics." If I understand correctly, your algos are real time adaptive and the process for adaptation is adaptive as well and parametric. I can not say it is good or not, but my modeling work in the last years has been focused on simplifying, so my question was: can't you make it simpler?
Last edited by chocolatemoney on December 8th, 2013, 11:00 pm, edited 1 time in total.