SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
chocolatemoney
Topic Author
Posts: 322
Joined: October 8th, 2008, 6:50 am

server for in-memory data storage

April 23rd, 2013, 12:15 pm

Hi,I have quite a big amount of TS data that I need to access for back-testing.I'd like to put together a web server where I could keep the whole mass of data loaded in the memory and serve the TS needed for the backtesting on the go.I was wondering if you attempted a similar project. I am still far from being confident in the choice of the technology stack.I'd like to write my code in Scala, hence I'd like to stick in the Java environment.My client will be a MATLAB application.- SQL or no-sql (mongodb)? What about H2 (http://www.h2database.com/html/main.html)- Is there some solution similar to Prevayler (http://prevayler.org/) designed for time series/arrays of floats?- REST or SOAP? Something specifically designed for arrays of floats?Thanks!
 
User avatar
chocolatemoney
Topic Author
Posts: 322
Joined: October 8th, 2008, 6:50 am

server for in-memory data storage

April 24th, 2013, 6:43 am

Thanks for your input. However, I am not sure I agree. The data required for the backtesting is just a tiny fraction of the total amount of data.With a dedicated machine, if I could process queries and cache tables.The skinny client (even a laptop), could just request the data needed and run the backtests. The server could also take care of some heavy-load calculations.In the future, the server machine will eventually take care of the backtests, a-z. This would be the ideal, final setup.Are there alternatives to mySQL you would consider?
 
User avatar
chocolatemoney
Topic Author
Posts: 322
Joined: October 8th, 2008, 6:50 am

server for in-memory data storage

April 25th, 2013, 11:59 am

I am programming in Scala.I have currently 40Gb, which is a fraction of the total database.The data needs to be consolidated but my guess is that the total will reach 1TbThe data is very heterogeneous, ranging from monthly data (some indices) to full order book.The backtest is left to the client.Real world execution runs through a different route, where order and positions net out each other and risk is monitored at "enterprise" level. This is something I will not be dealing with.I surely try and experiment, but I am kinda lost in the jungle of alternatives..Thanks again. EDIT: fixed a grammar error
Last edited by chocolatemoney on April 24th, 2013, 10:00 pm, edited 1 time in total.
 
User avatar
chocolatemoney
Topic Author
Posts: 322
Joined: October 8th, 2008, 6:50 am

server for in-memory data storage

May 2nd, 2013, 9:02 am

Yes, sure. Thanks for your input.Would you know a suitable java/c/c++ framework to manage a client-server connection for the exchange of binary structured data?Thanks again.
 
User avatar
chocolatemoney
Topic Author
Posts: 322
Joined: October 8th, 2008, 6:50 am

server for in-memory data storage

July 22nd, 2013, 7:52 am

Boost did the trick.The code is probably not "high level" as much as I wanted, and yes, it is C++ instead of a "friendlier" Python or Scala, but Boost did the trick.Thanks for your inputs!
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On