HiI am working in the algorithmic trading and arbitrage team as a tick-by-tick data analyst. Basically, I: -- Retrieve tick-data through ODBC from a tick database.-- Work with it using an analytical tool.-- Input it in another database using ODBC.Examples of output that I produce are distributions of: trade/quote frequencies, ADV and trading patterns. I also evaluate intra-day measures such as intra-day volatility. The software I am currently using is in-house and optimized for time series analysis and pricing for fixed income. It has support for business calendars and timestamps on the x-axis etc and is indeed convenient, however my gut-feeling is that it is not powerful enough when it comes to the statistic tools and optimization I intend to perform.What would be an optional software for me to evaluate? R, Matlab or S-plus? Anybody that has experience from producing graphs or analyzing tick-by-tick data using any of these packages. What toolboxes would be relevant?Extremely grateful for repliesRegardsMichael

QuoteOriginally posted by: LiKaShingHiWhat would be an optional software for me to evaluate? R, Matlab or S-plus? Anybody that has experience from producing graphs or analyzing tick-by-tick data using any of these packages. What toolboxes would be relevant?Any of R, Matlab or S-plus (which is jsut commercial R) can do the job. If money is no problem, my suggestion is matlab because it's easier to debug and also there are a lot of math packages with huge documentation.But, at the same time, I've seen people say that handling large data in matlab or R can be a pain. Perhaps C++ with some math library may be better, but I'm not sure.

LikaShing,I think it depends on how much tick data you are using for how many markets at the same time. I deal with tick data everyday however unlike using popular stat packages, I code optimization algos and statistical procedures in VB and excel. This way it not only allows me to modify the statistical procedures but find the optimal optimization algorithms and create very complicated trading logics all in one place with full control. I think it all comes down to what you are comfortable working with. We finish most of the coding within few hours and get the results if we had a defined plan ahead.

The problem with any standard client server protocol is that they don't cope well with large numbers of small data points.They basically work by sending a request to the server, waiting, then giving the data to the client application.Problem is that it takes quite a long time for each request to get to the server, processed and come back.OK "long" in this context might be 10 milliseconds, which is fine for most applications, but will be pretty tedious if you are dealing with serious time series.

Matlab (at least on 32-bit workstations) basically can't deal with datasets that occupy more than 300MB or so of RAM. If you can naturally write your code so that you never need more than that loaded at a time (eg process 1 day at a time and cache results) then I would recommend matlab, if not then use either C++ or whatever you're currently using to pre-chew the data and then use Matlab to postprocess it (there's a way to wrap C++ code as matlab native functions btw, or store intermediate data in text or hdf5 files). I would recommend Matlab because it's vastly better documented and has a vastly better GUI/debugger/code editor/command line etc. integration than either Splus or R. You'd want optimisation, statistics and database toolboxes, plus maybe some of the fancier analytical ones such as GARCH. I wouldn't bother with the financial toolbox - I didn't find anything in it that I couldn't replicate in 30 mins or so. On the other hand, Statistics toolbox has a lot of data mining stuff that you might use - PCA, decision trees, factor analysis, whatnot.The advantage of R (besides being free, which is irrelevant if you're at a commercial org) is really advanced statistical algos, but unless you already know what they are and how to use them, I don't see them doing you much good. And personally, I find the language itself a disaster, though many intelligent people disagree. Splus is, last I looked, a really expensive version of R with less functionality - but I may be wrong there, of course.

Is there a 64 bit version any time soon ?I've heard various comments about Matlab and issues with big arrays, and wonder if some Real Programming could be applied to it.Countless millenia ago, IBM gave me me money to work on expanding effective address ranges. It can be done, but I'd have to think more aobut Matlab arrays.That being said 32 bits ought to be enough for most quant jobs, even tick data series, if the the s/w is written right.

QuoteCountless millenia ago, IBM gave me mThat's not much to go on; what's the second letter?

Matlab does have 64 bit version that is supposed to help with the memory issues, I just didn't get around to playing with one yet. In any case, you could store the data from C++ in hdf5 and then use the random access capability of that format to read as much as you can handle at a time in matlab.

there are a couple of software companies that handle the tick data piece, some of them better than others. Kx systems kx.com is a APL derivative so if you're not scared of that its by far the most powerful out there. A more straightforward, but less capable product is from www.vhayu.com