SERVING THE QUANTITATIVE FINANCE COMMUNITY

 
User avatar
Cuchulainn
Posts: 59665
Joined: July 16th, 2004, 7:38 am
Location: Amsterdam
Contact:

Re: Parallel programming in Finance

September 1st, 2019, 9:25 am

I find the combination of FP and OOP optimal. In the boiler room we have OP and on top deck FP. 

Just out of curiosity; C# is very popular but I don't hear all that much about F#.
 
AndreyShihov
Topic Author
Posts: 15
Joined: August 12th, 2019, 2:38 pm

Re: Parallel programming in Finance

September 2nd, 2019, 2:04 pm

I find the combination of FP and OOP optimal. In the boiler room we have OP and on top deck FP. 

Just out of curiosity; C# is very popular but I don't hear all that much about F#.

I love the elegance of FP too, quite often mimic it in a small scale like Fluent Methods. Most of my experience it was OOP and SOLID for hottest bits (incl. data structures, tests, etc) as well. Business logic on MVC (and flavours) or DDD for large monoliths. I have seen some unsuccessful and very confusing implementations with functional approach, it is really easy to get things all around the house with it when designing framework. The code was very easy to read at the top level but number of Easter eggs that were hidden under those nicely named methods was mind blowing.  But I still like FP and want use it, specially try to play with F#.

I do design everything now with Microservices architecture in mind (containerisation and orchestration) to avoid scary hybrids that can be introduced by developers with different styles (functional, solid). The main benefits I see are: 

1) fun and diversity (experienced developers need it, want functional, here we go: service 1, 2, 3 pure F#) - can implement any service on any language or platform. Python, C++, F#, C#, C, Matlab, OOP, SOLID, functional, AI, ML, Linux, Win, Virtual, bare metal or whatever, as long as it satisfies requirements. That helps to keep developers' eyes peeled and most importantly it allows experts in a different subjects to contribute to the system independently.
2) versioning - this is most important. One can increase the version of the service and have both, old and new (or N), versions running until migration to the new version is complete. I have been extracting Microservices out of Monolith in the past and it works pretty well especially under DDD architecture. 

In my current project I have 3 containers: PDE (Closed form prices), MCS and QuantLib accessible via WebAPI to the clients such as SPA, WPF, Excel or Python. Each client can use them individually or parts of each. Each container will evolve independently and old versions will be online until they've been used by one of the clients. I would like to add ML container (python) and train it to predict (for instance) when system is likely to crash. What I've got: C#, C++, Python, TypeScript, XAML and all of that can evolve relatively independently. 
 
AndreyShihov
Topic Author
Posts: 15
Joined: August 12th, 2019, 2:38 pm

Re: Parallel programming in Finance

September 3rd, 2019, 10:32 am

OK, I have completed my experiments. I have used different data structures and algorithms. Memory profile looked like sawteeth or flat. CPU was around 100% or 50%; Had over 2M context switches or just 9K. All implementations complete simulation in about 40 sec. 

Expectation: I hoped avoidance of large memory allocations will improve performance.
Reality: For the setup I have mentioned earlier it actually made it worse. Path simulating tasks were always thirsty for small arrays (around 10000 elements) of random numbers and CPUs were running at around 70-80% (with around 17-20K context switches), although memory profile was flat. 

The overhead of LOH allocations was around 8%.

Solutions: 1) switch into unmanaged mode to have array allocations routine on fingertips (not recommended); 2) implement it in C++; 

Now I try to implement a random number generator on the other machine and get a buffer of random numbers to be used just for simulation on dedicated machine to see if it will improve performance by more than 8%. 
 
AndreyShihov
Topic Author
Posts: 15
Joined: August 12th, 2019, 2:38 pm

Re: Parallel programming in Finance

September 9th, 2019, 5:46 am

Also, forgot to add. In my earlier post I wasn't happy enough to see Monte-Carlo Simulation implemented with Blocking Collection I have seen in "C# book by Germani and Duffy". It turned out that my parallel implementation based on Blocking Collection is the winner in terms of speed. I have done it a bit different way - but anyway! I will base my future experiments on this implementation.
ABOUT WILMOTT

PW by JB

Wilmott.com has been "Serving the Quantitative Finance Community" since 2001. Continued...


Twitter LinkedIn Instagram

JOBS BOARD

JOBS BOARD

Looking for a quant job, risk, algo trading,...? Browse jobs here...


GZIP: On