Serving the Quantitative Finance Community
Storing Real-time Data
Say I have a Level 2 LSE feed and I wanted to store tick data directly from it....I know you can get tick data from a vendor but I was wondering if its plausible to store your own? Obviously for a large number of tickers it would be implausible in terms of storage space, but what about e.g. 10 UK large cap stocks over say a 1yr/255 day period?Could I store it if I build my own server or stick it on a RAID? Or am I completely bonkers?Appreciate the helpCheers
Storing Real-time Data
Seen this? https://github.com/kafka399/TwsMongo Don't know if it will eat quotes at the rate you'll be feeding, but I'm sure it's a good start. I'd say stay away from SQL for high-speed stuff, and go for one of the document-oriented ones. The stuff in the link use MongoDB.Btw, full disclosure: it's not my repo. S
Storing Real-time Data
Have you looked at kdb ?
Last edited by Rufus on August 2nd, 2012, 10:00 pm, edited 1 time in total.
-
- ghostrider
- Posts: 0
- Joined:
Storing Real-time Data
Over there at http://www.activequant.org we store years of tick data in our hbase+hadoop based solution. It's open core, so you can sort of use it for free up to a certain point (see the license for details). It is very blunt to store ticks as they arrive, that's why our feed recorder collects data for a certain time, say 5 seconds, and flushes these then. I have deployed our solution for several commercial clients - works flawless and reliable.