Anybody knows some good Matlab utilities for processing gigantic tick data set?I like the vectorized operations in Matlab and speed shouldn't be an issue;but the Matlab memory limitation is a problem...Any thoughts?

What kind of database are you using to store the ticks? How many time series do you need in memory at a time and how big are they (timescale, number and data type wise)?

If the dataset really is "gigantic" and memory is an issue, maybe Matlab isn't the best way to go, and you need to switch to C?

QuoteOriginally posted by: mizhaelAnybody knows some good Matlab utilities for processing gigantic tick data set?I like the vectorized operations in Matlab and speed shouldn't be an issue;but the Matlab memory limitation is a problem...Any thoughts?I've been working with 20gb+ data in matlab for my PhD. Here some suggestions:1) use a up to date computer with lots of ram.2) use proper database software (e.g. mySql) and load data in matlab by blocks. Don't forget to index the tables for speed.3) increase memory usage in matlab. You can find instructions on google.4) If you can't do number 1), install and run http://www.iobit.com/gamebooster.html when running the codes. Hope it helps..

QuoteOriginally posted by: frattyquantIf the dataset really is "gigantic" and memory is an issue, maybe Matlab isn't the best way to go, and you need to switch to C?I think Matlab's vectorized operation would be ideal. The only problem is the memory. But I guess people handle data in trunks anyway.So I am looking for packages/toolboxes that can help HFT ...

QuoteOriginally posted by: HansiWhat kind of database are you using to store the ticks? How many time series do you need in memory at a time and how big are they (timescale, number and data type wise)?What are good databases to use for HFT?Matlab can handle all kinds of database using Database toolbox...I am thinking of storing data in KDB and using Matlab to operate on the data ...Anybody had experiences?

There are a few threads over in Software on databases. The options I'd recommend would be BerkleyDB, KDB or HDF5. Berkley is most likely hardest to manage but has excellent speed. HDF5 is easy to manage and relatively fast. KDB+ is okay but a bit expensive.But note that Matlab is not an option for HFT, it's way to slow and has too much overhead. It's decent for prototyping, back testing etc. for trading strategies but it's not going to be fast enought for HFT. For small shop medium frequency trading (and general algorithmic trading) it will be fine (<10 bulk trades intra day or maybe every 2-3 days). For non co-located HFT you will most likely want C++ with proper memory managed access to read the ticks and work with them in real time (but you'll be at a huge disadvantage even then because of not being co-located). For co-located stuff you're best of getting advice from people with experience in that space since I have none.

hey mizhael... just saw your similiar post on ET....matlab is great for vector math, the problem is getting large amounts of data in and out of matlab... both in live and historical/backtesting situations.what a lot of people (including myself do), is use an external tool to run simulations really fast and then use R or matlab to interpret and also do optimization of results.

It might not be a bad idea to pick up a perl book. Any scripting language would be good for this task, even sql can do what you ask.