Been reviewing designs for an intraday trading platform - prefer Python based for the algos. Need to run a few hundred agents in parallel just during the day - fed by KAFKA topics. The agents basically calculate the data frame and perform some methods to evaluate a signal, fire a signal if the algo hits. We have been reviewing some designs for this - like to get some advice from the folks here? Needs to scale horizontally - cluster.
Latency few hundred milliseconds is fine.
Dask: to run agents concurrently in python, will this work, not sure the use case matches dask?
Akka: using java actor framework bundled with python to execute?
Erlang/Elixir: use erlang actor framework bundled with python?
Spark: use spark to run many spark jobs
Not sure which way to go, like to keep all python for ease of maintainability - any advice appreciated.. The trick is link with python - the algo libraries..