Does anyone have firsthand experience with python for data exploration/prototyping/playing around with ideas (especially on datasets >> 100MB)? I've used matlab so far, but its inability to cope with mildly large datasets is starting to really annoy me (I give the VM 1Gb of RAM, and it still chokes on datasets of around 150Mb).From a first glance, it seems that the python language structure has most of the features I like in matlab (dynamic; concise; good debugger; hashes, functions, primitives and classes are first-class objects), and NumPy/SciPy would give most of the linear algebra etc. functionality that I want. Also, eclipse + PyDev Extensions look like an IDE almost equal to that of matlab, with decent debugging, code completion, etc. Are there any snags in switching to the above combo from matlab, that are not obvious before one tries? I remember hearing somewhere that one IB was using python a lot - any idea which?

QuoteOriginally posted by: ZmeiGorynychDoes anyone have firsthand experience with python for data exploration/prototyping/playing around with ideas (especially on datasets >> 100MB)? I've used matlab so far, but its inability to cope with mildly large datasets is starting to really annoy me (I give the VM 1Gb of RAM, and it still chokes on datasets of around 150Mb).From a first glance, it seems that the python language structure has most of the features I like in matlab (dynamic; concise; good debugger; hashes, functions, primitives and classes are first-class objects), and NumPy/SciPy would give most of the linear algebra etc. functionality that I want. Also, eclipse + PyDev Extensions look like an IDE almost equal to that of matlab, with decent debugging, code completion, etc. Are there any snags in switching to the above combo from matlab, that are not obvious before one tries? I remember hearing somewhere that one IB was using python a lot - any idea which?I've never spent that much time with matlab, but python is great for numeric analysis with some caveats. Not sure how far >> than 100MB you want to go, but I highly recommend having 2-4x physical ram over the data set to allow for fragmentation and temp arrays when using say numpy. I prefer to keep large datasets in numpy arrays to avoid per datum memory overheads. Given 64 bit machine and plenty of RAM then you can prototype very quickly. You can use weave.blitz to get rid of the temps and get pretty big speedups. matplotlib is great as well for plotting. Its not that hard to hack something up to use mmap backed arrays if you want to use datesets > physical RAM. Depending on how big the data you are wanting to play with, pytables might be useful for you.R is great as well btw.

No technical insights here, but a guy I know is a big fan of Python for rapid development in finance. Powerful statistical and scientific libraries exist and are easy to get. (Diagonalize large matrices no problem). Large community of users. He did a lot of slicing and dicing of data for me, and it seemed to serve him well.

I use python extensively to analyse datasets in excess of 100Mb. I also use R a fair bit. In my experience - they have different purposes. I tend to use pthon as a second stage anaysis/development platform. There are also two projects that actually allow interop between python and R - worth checking out, especially given you have experience with R already.

QuoteOriginally posted by: PaperCutNo technical insights here, but a guy I know is a big fan of Python for rapid development in finance. Any chance that guy might want to exchange an email or two with me?Quote Originally posted by: afosterI use python extensively to analyse datasets in excess of 100Mb. How much in excess? When does data size start being a problem? Also, what toolchain would you recommend?Thanks a lot,E.

Quotewhat toolchain would you recommend?What is a toolchain, I have never heard before?

QuoteOriginally posted by: CuchulainnWhat is a toolchain, I have never heard before?Complementary set of tools, often used one after the other, to produce software. eg. vim, gcc, ddd, svn. Some prefer IDEs with support for the various tools, others prefer a set of independent tools. For python there are a number of options in free and commercial IDEs, and its well supported by the common choices of vim and emacs as well.I suppose for python I shouldn't assume the cpython implementation is the one of choice necessarily.. as ironpython for .net and jython for java are also available.

QuoteHow much in excess? When does data size start being a problem?200Mb is the max I looked at, my machine is a dual core intel something or rather with 2Gb RAM. R essentially keeps all objects in memory - so the performance is dependant on how many objects you create. Python can get really slow with 50Mb + data dependant on your program design. If you cash hundreds of tiny objects then you will suffer from poor performance, but this is true in any language. Using numpy or even standard arrays is straightforward and painfree. Don't forget - its all c code underneath!QuoteAlso, what toolchain would you recommend?I use python (2.4) - and a few relevant libs such as numpy, fpconst, rpy. For editing, I use the in-built IDLE GUI, as eclipse is just so heavyweight I find it an overkill. Database wise I use PostgreSQL 8.1.4, with mxODBC as my python interface. I have written programs that synchronise this data with Bloomberg (in python). My R install is 2.3.1 with various libs such as fSeries, fBasics and a database connection lib.

QuoteOriginally posted by: ZmeiGorynych QuoteOriginally posted by: PaperCutNo technical insights here, but a guy I know is a big fan of Python for rapid development in finance. Any chance that guy might want to exchange an email or two with me?Sorry. I've moved on from that firm...

I recently (within the last 6-9 months) moved out of academia (finally) into the real world. My new group (to be kept close to the vest...sorry) has a keen interest in Python as a development tool (both as a "glue" language for legacy and new compiled codes and for pure development of apps using NumPy/SciPy and other packages). I was wondering what people think about what Python's status is among various development environments in QF, and what people think about it as a MATLAB/Mathematica/R/etc. other desktop tool alternative as well as alternative to various "compiled" languages that I could invest my time in developing? I know everything in life is application dependent, but where is Python going in QF and should I spend my time there? Thanks in advance for the thoughts and opinions.

I like Python and NumPy, but one defect IMO is the coming instability of the language. Python 3 will NOT be backward-compatible with Python 2.x, and even some basic syntax suchprint "hello world"will be removed to make "print" a function. The Python developers will provide scripts to convert Python 2.x code to Python 3 code, but I think it is fundamentally wrong for a language designer to "fix what isn't broken".I don't think one needs to worry about designers breaking backward compatibility as much with alternatives such as Matlab or R.

Quote Python 3 will NOT be backward-compatible...The wikipedia page on Python 3.0 has quite a good explaination of this statement and more. Seeing as Python 3.0 won't even be available for at least a year from now, and that Python 2.x will still continue to be supported and developed, and that there will be migration tools - is this really an issue?

QuoteOriginally posted by: ZmeiGorynychFrom a first glance, it seems that the python language structure has most of the features I like in matlab (dynamic; concise; good debugger; hashes, functions, primitives and classes are first-class objects), and NumPy/SciPy would give most of the linear algebra etc. functionality that I want. Also, eclipse + PyDev Extensions look like an IDE almost equal to that of matlab, with decent debugging, code completion, etc. The big thing that is missing is an object that resembles data frames in R. There is one on the SciPy wiki, and I've modified it to include more functionality. At some point I really need to upload it to the SciPy wiki.

QuoteOriginally posted by: abginflI know everything in life is application dependent, but where is Python going in QF and should I spend my time there? Thanks in advance for the thoughts and opinions.Python makes a very nice glue language. People have put a lot of thought into the CS aspects of the language (unlike R) and it is open source (unlike mathematica). The big advantage of Python is that there is already a very well developed scientific programming community with python that QF folks can leverage off of.

What are data frames in R, and what are they good for? I thought they were kind of like structs.