Is xarray to be preferred over ndarray?Maybe this?Can you recommend an open binary format to store large simulation output consisting of columns of numbers and strings? (I can get rid of strings if necessary.)
Serving the Quantitative Finance Community
-
- Cuchulainn
- Posts: 22933
- Joined:
Re: Open binary file format
-
- Cuchulainn
- Posts: 22933
- Joined:
Re: Open binary file format
Hey, let's say you wanted to solve the 3d heat equation numerically. Could HDF5 be used
1. In 'real-time' on a machine (awful slow?)
2. Off line training and then store the HDF5 data for later generalisation?
xarray + NetCDF looks better.
1. In 'real-time' on a machine (awful slow?)
2. Off line training and then store the HDF5 data for later generalisation?
xarray + NetCDF looks better.
-
- FaridMoussaoui
- Posts: 327
- Joined:
- Location: Genève, Genf, Ginevra, Geneva
Re: Open binary file format
Until you do the experiments, can't conclude.
Bethel and al, analysed the 2010 Flash Crash using HDF5 as their internal data format with GB of data on HPC systems.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1939522
As usual, you can get the document throught sci-hub.
Bethel and al, analysed the 2010 Flash Crash using HDF5 as their internal data format with GB of data on HPC systems.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1939522
As usual, you can get the document throught sci-hub.