Serving the Quantitative Finance Community

 
User avatar
ashkar
Topic Author
Posts: 0
Joined: October 17th, 2011, 9:25 am

Software engineering in python

March 11th, 2016, 6:16 pm

Around 3months ago, I started writing a library in python that calls into my c++ library for pricing deals. The python library is a wrapper to do useful things with a simpler interface (e.g instead of 5 lines of c++ code, the python interface will give a timeseries in 1 line of code). Initially I found it great, my productivity was much higher. After 3 months of writing python I'm beginning to question python in terms of software stability (wrt bugs and performance). Maybe I'm not using python in the correct way.For example, when I picked up python, I would write interface for all trade types etc as one would do in a strongly typed language...then I stopped doing it because its not pythonic. I went with the flow reading on stack overflow etc... Now my library is slightly bigger and the thought of making changes to any of the main classes just scares me since I cant compile the code. I asked one seasoned python developers what they do about it and his answer was to write many unit tests. Ok. So I thought maybe test-driven approach is what I should be doing for python. So I have started to write unit tests to trigger all the conditions in the code but this is really painful. The tests have effectively become my way of ensuring that the python codebase has no syntax/interpreter errors!!!Now I'm contemplating going back to c++. I've had my adventures in python and it was fun but I don't feel confident developing libraries in this language.Am I missing something? Maybe I need to adapt my software engineering methods when writing libraries in python? Or just need more experience writing python?
 
User avatar
Hansi
Posts: 41
Joined: January 25th, 2010, 11:47 am

Software engineering in python

March 12th, 2016, 10:58 am

Nope your thoughts pretty much mirror my experience, giving up the statical typing and going with dynamic (/duck) typing means you'll need to write more defensively and unit test quite a bit.But unit testing a library extensively is pretty much what you have to do to ensure it's accuracy when re-factoring.Note there is a library in Python that adds statical typing: http://mypy-lang.org/We've had this same discussion back and forth in terms of R vs C++, C# or Java at my firm and we decided to stick with R and use RCpp where applicable to ensure productivity and ease ofuse + package access but damn it making large changes without sufficient unit and integration tests is nigh impossible so the logic is to upskill testing before making large changes.
 
User avatar
Cuchulainn
Posts: 20255
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Software engineering in python

March 12th, 2016, 12:29 pm

QuoteNow I'm contemplating going back to c++. I've had my adventures in python and it was fun but I don't feel confident developing libraries in this language.Python is the new Matlab.No free lunch I suppose. Can you name one dynamically typed language that is used in production software?
Last edited by Cuchulainn on March 11th, 2016, 11:00 pm, edited 1 time in total.
 
User avatar
dweeb
Posts: 11
Joined: July 11th, 2009, 8:10 pm

Software engineering in python

March 12th, 2016, 4:59 pm

Thx for all the very useful information. I started dipping my toe into the Python world, but I'll take Hansi's route as I'm comfortable with R and C.
 
User avatar
Hansi
Posts: 41
Joined: January 25th, 2010, 11:47 am

Software engineering in python

March 12th, 2016, 7:25 pm

QuoteOriginally posted by: CuchulainnCan you name one dynamically typed language that is used in production software?Yes most of them are used in production somewhere Python and Ruby are the most common ones on the Web front and R, SAS and Python are big for big data analytics.We have a bunch of R code being used in production for screening, optimizations, analytics etc.Have a look at some of the stacks being used by medium to large tech companies here: http://stackshare.io/stacks
 
User avatar
ashkar
Topic Author
Posts: 0
Joined: October 17th, 2011, 9:25 am

Software engineering in python

March 13th, 2016, 11:51 am

Thanks Hansi. Lots of useful pointers there. I havent got too far with python yet so I can still move the core parts of the python library back into c++. My impression now is to keep python as a very light layer utilising pandas, numpy etc. I have known about static compilation packages in python but decided not to use it since a lot of open source python libraries die after few years.My backend code is almost ready. The next stage for me is to do something interesting with the data so I have the choice of python and R. I dont know what statistical methods and optimisation methods I will need so I want to explore bits of both. I use swig to generate c++ wrappers for python. It can also generate wrappers for R (and many other languages) by just changing the target language in the command. Did you guys look into this before going with rcpp?
 
User avatar
ashkar
Topic Author
Posts: 0
Joined: October 17th, 2011, 9:25 am

Software engineering in python

March 13th, 2016, 12:04 pm

QuoteOriginally posted by: CuchulainnQuoteNow I'm contemplating going back to c++. I've had my adventures in python and it was fun but I don't feel confident developing libraries in this language.Python is the new Matlab.No free lunch I suppose. Can you name one dynamically typed language that is used in production software?Quite right. For me, python with ipython notebook/jupyter is a replacement for Excel/vba with more expressive language and access to many data/statistical libraries. One of the key advantage of python/notebook environment is that the data visualsation can be made interactive using python/javascript libraries (e.g plotly). Matlab is better than Excel/R in that respect but its not free.I hear BAML's core library is in python and other banks also use python or other dynamically typed languages however I dont know how production those things are.
 
User avatar
Cuchulainn
Posts: 20255
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Software engineering in python

March 13th, 2016, 3:07 pm

QuoteAm I missing something? Maybe I need to adapt my software engineering methods when writing libraries in python? Or just need more experience writing python?Hard to say really, but you are not the first developer in the history of computing who has run into maintenance issues. Many are caused by 1. Short term positive results and over-optimism evolving into a quagmire of maintenance problems.2. No upfront design .. jump straight into coding (which managers like).3. Is Python only suitable for one-off throwaway prototypes?4. The "get it working, then get it right, then get it optimized" cycle is not very pervasive (for many technical, organizational and hum reasons).5. No design 'with maintenance in mind'.But it might be different in the current case. No idea.
Last edited by Cuchulainn on March 12th, 2016, 11:00 pm, edited 1 time in total.
 
User avatar
Traden4Alpha
Posts: 3300
Joined: September 20th, 2002, 8:30 pm

Software engineering in python

March 13th, 2016, 3:29 pm

QuoteOriginally posted by: CuchulainnQuoteAm I missing something? Maybe I need to adapt my software engineering methods when writing libraries in python? Or just need more experience writing python?Hard to say really, but you are not the first developer in the history of computing who has run into maintenance issues. Many are caused by 1. Short term positive results and over-optimism evolving into a quagmire of maintenance problems.2. No upfront design .. jump straight into coding (which managers like).3. Is Python only suitable for one-off throwaway prototypes?4. The "get it working, then get it right, then get it optimized" cycle is not very pervasive (for many technical, organizational and hum reasons).5. No design 'with maintenance in mind'.But it might be different in the current case. No idea.Those are all good reasons. It's like when desktop publishing first arrived and everyone created atrocious documents with ransom-note font choices.'Easy to use is easy to abuse.
 
User avatar
Cuchulainn
Posts: 20255
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Software engineering in python

March 13th, 2016, 3:57 pm

QuoteOriginally posted by: Traden4AlphaQuoteOriginally posted by: CuchulainnQuoteAm I missing something? Maybe I need to adapt my software engineering methods when writing libraries in python? Or just need more experience writing python?Hard to say really, but you are not the first developer in the history of computing who has run into maintenance issues. Many are caused by 1. Short term positive results and over-optimism evolving into a quagmire of maintenance problems.2. No upfront design .. jump straight into coding (which managers like).3. Is Python only suitable for one-off throwaway prototypes?4. The "get it working, then get it right, then get it optimized" cycle is not very pervasive (for many technical, organizational and hum reasons).5. No design 'with maintenance in mind'.But it might be different in the current case. No idea.Those are all good reasons. It's like when desktop publishing first arrived and everyone created atrocious documents with ransom-note font choices.'Easy to use is easy to abuse.If there is standardization then you work with standard interfaces instead of umpteen home-grown ones. Of course, this is easier said than done. See my point #4.
Last edited by Cuchulainn on March 12th, 2016, 11:00 pm, edited 1 time in total.