Serving the Quantitative Finance Community

 
User avatar
katastrofa
Topic Author
Posts: 7440
Joined: August 16th, 2007, 5:36 am
Location: Alpha Centauri

Re: Open binary file format

April 10th, 2019, 8:07 pm

Thank you, Farid. I'm only asking for your opinion if you have spare time and are keen on cracking others' CraPPy codes :-)
Re seconds_t, it's just a redefinition of the double type.
 
User avatar
rmax
Posts: 374
Joined: December 8th, 2005, 9:31 am

Re: Open binary file format

April 24th, 2019, 12:18 pm

Depends on the use case. KDB? not a binary format but perhaps you need more than that...
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Open binary file format

May 2nd, 2019, 2:04 pm

Can you recommend an open binary format to store large simulation output consisting of columns of numbers and strings? (I can get rid of strings if necessary.)
Maybe this?

Is xarray to be preferred over ndarray?
 
User avatar
Cuchulainn
Posts: 20254
Joined: July 16th, 2004, 7:38 am
Location: 20, 000

Re: Open binary file format

October 7th, 2019, 6:56 pm

Hey, let's say you wanted to solve the 3d heat equation numerically. Could HDF5 be used

1. In 'real-time' on a machine (awful slow?)
2. Off line training and then store the HDF5 data for later generalisation?

xarray + NetCDF looks better.
 
User avatar
FaridMoussaoui
Posts: 327
Joined: June 20th, 2008, 10:05 am
Location: Genève, Genf, Ginevra, Geneva

Re: Open binary file format

October 7th, 2019, 10:58 pm

Until you do the experiments, can't conclude.

Bethel and al, analysed the 2010 Flash Crash using HDF5 as their internal data format with GB of data on HPC systems.

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1939522

As usual, you can get the document throught sci-hub.