QuoteOriginally posted by: ZmeiGorynychWhat are data frames in R, and what are they good for? I thought they were kind of like structs.The are sort of like spreadsheets. You have tabular data, but you can refer to a column by a name and you can attach attributes to a column and the table as a whole. Most of the plotting and regression routines can take in data frames so once you have your data in one, it's easy to do plots and regressions with them.

QuoteOriginally posted by: ZmeiGorynychDoes anyone have firsthand experience with python for data exploration/prototyping/playing around with ideas (especially on datasets >> 100MB)? I've used matlab so far, but its inability to cope with mildly large datasets is starting to really annoy me (I give the VM 1Gb of RAM, and it still chokes on datasets of around 150Mb).I noticed today that Interactive Supercomputing is providing download evaluation copies of Star-P for Python at no cost. I used their Matlab version in grad school, and obtained some very good results. I think I am going to have to find some spare cycles on the servers at work to play with this Python version. If anyone else on here tries it out, then let me know what you think of their Python product.

I personally dont like Python the language, and I know a lot of other people who feel the same way. However, it does seem to have a decent following for this kind of stuff - I know some quants who use it for prototyping their models, for instance.

What about PyTables built on top of HDF for data storage? It seems to me that it can handle really huge datasets really well.Data I'm working is rather small, but i'm in desperate need of a good interface.Tired of dirty scripts dealing with shelves...

Interesting old thread... I use Python essentially exclusively where people use matlab and similar packages. Some comments:I've used PyTables and found them to very good. But the main use cases for them is if you have data with a lot of structure, need to archive it permanently or distribute widely. If you just have a few big arrays to dump to disk you may as well do this directly.Regarding large data structures, I've had no problems handling 8GB arrays on machines with 16GB of RAM. Somebody mentioned memory mapped files as a way of dealing with truly large data sets. This is supported by numarray and numpy out of the box. Just do numpy.load("foo", memmap=True)You can access most (all?) functionality of quantlib through the python binding.Or if you are developing your own algorithms, it is very easy to expose them for use in python. I use SWIG but other choices are boost, f2py etc

I use both python and matlab extensively at my work (commodity quant@energy trading firm).My personal feeling is that python as a language is a much better designed and thought out piece of software than matlab is. However, I prefer to use matlab for most of work for the following reasons:1. IDE - matlabs work environment is clean and polished allowing me to be very productive. I have spent many hours trying to duplicate the same functionality with open source tools and have been unsuccessful... i have tried: emacs/vim/idle/komodo/eclipse/matplotlib/ipython... nothing is as good, if i have missed anything please let me know.2. Documentation - matlab's documentation is really very good.3. Toolboxes. Python has a lot of overlap with these, but matlab has a lot more, well designed packages that work together nicely.The downside of course is the price... I think that the open source arguement doesnt hold as much water in this case as most of the library functions/toolbox functions in matlab are implemented in M-code to which you have the full source to...just my 2cents.

I'm looking at this thread with much interest. I'm starting up as a quant with a energy trading firm, and I guess that I'll run into some of the issues you are discussing in the future. I have previously used Python and Fortran for some FDM for my masters thesis, and really enjoy Python. At this point my main concerns are:* Pythons abilities when it comes to working with Excel. I am aware of the win32 module, but how does it perform? * Anybody use Python to develop utilities for the front desk? Really, could I expect to get to use Python, or should I just start learning VB right now?Regards,Stale

spmacdonald, I agree with you completely regarding documentation, it is a big barrier to using many of the python packages... or even figuring out which package to use. People often get stuck just deciding between numeric/numarray/numpy. I typically end up only ever using the online help and reading the source code.The main reason Python wins in my book is that all those little scripts end up being used again and again, and by a number of different people. At that point you need to transition from quickly written script to a well engineered and long-lived one and Python supports that very well.As for IDEs I use emacs and it suits me perfectly, but then I again, I write code for MS Visual c++ etc in emacs too. For more closely replicating the type of environment matlab provides, I'd have a look at SAGE:http://www.sagemath.org/

spammcdonald,WingIDE. It's not a free software, but is definetly worth trying. I got it at work and am really happy with it.And ipython with scipy is really nice, isn't it?matplotlib... Actually I quite dislike pylab interface which almost resembles the one from matlab.Would like to create something reusable around this library, but it doesn't seem that easy.

QuoteOriginally posted by: eliomatplotlib... Actually I quite dislike pylab interface which almost resembles the one from matlab.Would like to create something reusable around this library, but it doesn't seem that easy.Same here. The only interactive alternative I know is Chacohttp://code.enthought.com/chaco/but haven't looked at that in a long while. Anybody has experience with it?

here's another interactive plotting libhttp://home.gna.org/veusz/

I looked both at veusz and chaco. Both are worth looking at.veusz seems a small nice thing. it has a decent GUI where u cantweak the plot manually, similarly to MATLAB. One-man project,documentation is quite brief.Chaco is a heavy machinery. Looks like a properly designed plottinglibrary with a focus on interaction with the user.At first I disliked it while looking at the annotated examples (so many lines for a small plot), but actually the code is readable andthe library seems to be a really flexible one. Interactivity is supported.

There is also a project to wrap r with python. RPy is a little young yet, but worth monitoring - could be very interesting.

rpy works well, the documentation's not so good though.(i used it to do quite basic stuff though).

RPy is ok, but I find it easier to build my datasets in python, then simply save them to csv and analyse them in R seperately. For editing, I use e text editor, which is very similar to TextMate.