Serving the Quantitative Finance Community

 
crunchdata
Topic Author
Posts: 1
Joined: May 14th, 2018, 1:20 pm

Dask - Parallel Trading agents Horizontal Scale

May 14th, 2018, 1:25 pm

Hello, 

Been reviewing designs for an intraday trading platform - prefer Python based for the algos.  Need to run a few hundred agents in parallel just during the day - fed by KAFKA topics.  The agents basically calculate the data frame and perform some methods to evaluate a signal, fire a signal if the algo hits.   We have been reviewing some designs for this - like to get some advice from the folks here?   Needs to scale horizontally - cluster.  

Latency few hundred milliseconds is fine. 

Dask:  to run agents concurrently in python, will this work, not sure the use case matches dask? 
Akka:  using java actor framework bundled with python to execute?
Erlang/Elixir:  use erlang actor framework bundled with python? 
Spark:  use spark to run many spark jobs 

Not sure which way to go, like to keep all python for ease of maintainability - any advice appreciated..   The trick is link with python - the algo libraries.. 
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Dask - Parallel Trading agents Horizontal Scale

May 15th, 2018, 7:18 am

My 2 cents

I have ported an OO MC pricer using system deccomposition to two actor .NET actor systems Asynchronous Agent Library (C++) and  then to TP:L Dataflow Library whose goal seems to be low latency. The functionality looks good but I have no idea how it would scale. These libraries are for multi-core machines..


http://blogs.microsoft.co.il/bnaya/2011 ... ow-part-1/

Needs to scale horizontally - cluster.  
You mean more dynamic consumers and producers and algorithms?

The original OO plan is in the pdf. The actor model is more suitable here.
Attachments
MC Application.pdf
(953.93 KiB) Downloaded 130 times
 
User avatar
outrun
Posts: 4573
Joined: January 1st, 1970, 12:00 am

Re: Dask - Parallel Trading agents Horizontal Scale

May 15th, 2018, 8:23 am

If you want to stick with python then you can manage and deploy your algos with flask -a popular framework to build python webservices http://flask.pocoo.org/-, then run them as services via any of the popular webservers or standalone http://flask.pocoo.org/docs/0.12/deploying/

Linking thinks between processes or across machines up can easily be done in any language with (web)sockets and some data serialization protocol.