SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
mizhael
Topic Author
Posts: 623
Joined: September 25th, 2005, 4:46 pm

Pointers to good Matlab utilities for processing gigantic tick data set?

June 1st, 2010, 6:53 pm

Anybody knows some good Matlab utilities for processing gigantic tick data set?I like the vectorized operations in Matlab and speed shouldn't be an issue;but the Matlab memory limitation is a problem...Any thoughts?
 
User avatar
Hansi
Posts: 3300
Joined: January 25th, 2010, 11:47 am

Pointers to good Matlab utilities for processing gigantic tick data set?

June 1st, 2010, 8:33 pm

What kind of database are you using to store the ticks? How many time series do you need in memory at a time and how big are they (timescale, number and data type wise)?
 
User avatar
frattyquant
Posts: 362
Joined: March 4th, 2010, 8:10 am

Pointers to good Matlab utilities for processing gigantic tick data set?

June 2nd, 2010, 7:11 am

If the dataset really is "gigantic" and memory is an issue, maybe Matlab isn't the best way to go, and you need to switch to C?
 
User avatar
msperlin
Posts: 608
Joined: July 10th, 2006, 6:21 pm

Pointers to good Matlab utilities for processing gigantic tick data set?

June 2nd, 2010, 9:07 am

QuoteOriginally posted by: mizhaelAnybody knows some good Matlab utilities for processing gigantic tick data set?I like the vectorized operations in Matlab and speed shouldn't be an issue;but the Matlab memory limitation is a problem...Any thoughts?I've been working with 20gb+ data in matlab for my PhD. Here some suggestions:1) use a up to date computer with lots of ram.2) use proper database software (e.g. mySql) and load data in matlab by blocks. Don't forget to index the tables for speed.3) increase memory usage in matlab. You can find instructions on google.4) If you can't do number 1), install and run http://www.iobit.com/gamebooster.html when running the codes. Hope it helps..
 
User avatar
mizhael
Topic Author
Posts: 623
Joined: September 25th, 2005, 4:46 pm

Pointers to good Matlab utilities for processing gigantic tick data set?

June 2nd, 2010, 11:06 am

QuoteOriginally posted by: frattyquantIf the dataset really is "gigantic" and memory is an issue, maybe Matlab isn't the best way to go, and you need to switch to C?I think Matlab's vectorized operation would be ideal. The only problem is the memory. But I guess people handle data in trunks anyway.So I am looking for packages/toolboxes that can help HFT ...
 
User avatar
mizhael
Topic Author
Posts: 623
Joined: September 25th, 2005, 4:46 pm

Pointers to good Matlab utilities for processing gigantic tick data set?

June 2nd, 2010, 11:08 am

QuoteOriginally posted by: HansiWhat kind of database are you using to store the ticks? How many time series do you need in memory at a time and how big are they (timescale, number and data type wise)?What are good databases to use for HFT?Matlab can handle all kinds of database using Database toolbox...I am thinking of storing data in KDB and using Matlab to operate on the data ...Anybody had experiences?
 
User avatar
Hansi
Posts: 3300
Joined: January 25th, 2010, 11:47 am

Pointers to good Matlab utilities for processing gigantic tick data set?

June 2nd, 2010, 11:40 am

There are a few threads over in Software on databases. The options I'd recommend would be BerkleyDB, KDB or HDF5. Berkley is most likely hardest to manage but has excellent speed. HDF5 is easy to manage and relatively fast. KDB+ is okay but a bit expensive.But note that Matlab is not an option for HFT, it's way to slow and has too much overhead. It's decent for prototyping, back testing etc. for trading strategies but it's not going to be fast enought for HFT. For small shop medium frequency trading (and general algorithmic trading) it will be fine (<10 bulk trades intra day or maybe every 2-3 days). For non co-located HFT you will most likely want C++ with proper memory managed access to read the ticks and work with them in real time (but you'll be at a huge disadvantage even then because of not being co-located). For co-located stuff you're best of getting advice from people with experience in that space since I have none.
Last edited by Hansi on June 1st, 2010, 10:00 pm, edited 1 time in total.
 
User avatar
tradelink
Posts: 55
Joined: March 9th, 2010, 9:55 pm

Pointers to good Matlab utilities for processing gigantic tick data set?

June 2nd, 2010, 1:28 pm

hey mizhael... just saw your similiar post on ET....matlab is great for vector math, the problem is getting large amounts of data in and out of matlab... both in live and historical/backtesting situations.what a lot of people (including myself do), is use an external tool to run simulations really fast and then use R or matlab to interpret and also do optimization of results.
 
User avatar
KKleinops
Posts: 4
Joined: June 29th, 2010, 5:58 pm

Pointers to good Matlab utilities for processing gigantic tick data set?

July 7th, 2010, 6:47 pm

It might not be a bad idea to pick up a perl book. Any scripting language would be good for this task, even sql can do what you ask.
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On