Page 18 of 21

Re: Python tricks

Posted: June 15th, 2020, 9:30 pm
by ISayMoo
Any tips how to write a simple `setup.py` for the binary .pyd file?

Re: Python tricks

Posted: June 16th, 2020, 8:38 am
by Cuchulainn
Any tips how to write a simple `setup.py` for the binary .pyd file?
We did some initial tests a few months back but I need to revisit it again as we shall  wrap (rough) Heston code  in Python.

The setup used was
import os, sys

from distutils.core import setup, Extension
from distutils import sysconfig

cpp_args = ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7']

sfc_module = Extension(
    'PybindDLL', sources = ['module.cpp'],
    include_dirs=['pybind11/include'],
    language='c++',
    extra_compile_args = cpp_args,
    )

setup(
    name = 'PybindDLL',
    version = '1.0',
    description = 'Python package with superfastcode2 C++ extension (PyBind11)',
    ext_modules = [sfc_module],
)

Re: Python tricks

Posted: June 16th, 2020, 10:12 pm
by ISayMoo
I'm building the .pyd file in Visual Studio.

Re: Python tricks

Posted: June 17th, 2020, 9:46 pm
by ISayMoo
OK, solved it.
from setuptools import setup, find_packages

setup(
    name="PyML",
    version="0.1",
    packages=find_packages(),
    include_package_data=True,
    zip_safe=False,
    package_data={"PyML": ["PyML.pyd", "ML.dll"]}

Re: Python tricks

Posted: June 18th, 2020, 8:54 am
by Cuchulainn
Cool. Is this public domain and can we use it for our ML project? 

I tried a number of binders already, by hand. The biggest pain was making that dll 'visible' to the ciient code.

Re: Python tricks

Posted: June 24th, 2020, 7:26 pm
by Cuchulainn
I'm building the .pyd file in Visual Studio.
"Thanks a lot. It is helpful.

Another question arises, what if I have multiple sources file?

All of the examples I can find online are for one source file. I am struggling to make it work for project that contains multiple .cpp files."

Re: Python tricks

Posted: June 25th, 2020, 7:20 pm
by ISayMoo
I'll post the code some time soon. It will be Open Source.

Re: Python tricks

Posted: June 26th, 2020, 9:56 am
by Cuchulainn
The goal of the project is a mix of C++ and Python, narrow (replaceable) interfaces and a defined/standardised data flow. C++11 futures and GPU if time permits. A bit like kanban boards.
This is why pybind11 will be useful.
And project risk mitigation (budget == 3 months).
Image

Re: Python tricks

Posted: June 27th, 2020, 7:42 pm
by ISayMoo
Interesting.

Re: Python tricks

Posted: July 11th, 2020, 11:53 am
by Cuchulainn
What's the best/easiest way to expose C++ std::vector and std::vector<std::vector> (matrix) as equivalent Python data structures?

Re: Python tricks

Posted: July 11th, 2020, 12:22 pm
by katastrofa
To what structure? List or array (converting vector of vectors to it)?

Or do you mean sharing access to the same memory? You're writing some ugli codez there... ?

Re: Python tricks

Posted: July 11th, 2020, 1:52 pm
by Cuchulainn
From C++ vectors/"matrices" to lists/arrays of tuples would  be fine, for starters. I haven't the time for an odyssey into all the options as it is extremely time-consuming at this stage.

Re: Python tricks

Posted: July 11th, 2020, 8:14 pm
by katastrofa

Re: Python tricks

Posted: July 19th, 2020, 11:50 am
by Cuchulainn
This terrein is like being the in the Empty Quarter!
Long story short .. It took one of us 2 weeks to get how it worked (eventually finding the details from a Chinese web site..)
pybind11 has marshalling interface to Eigen library. But for other matrix types

1. Write interface to Eigen, easy?
2.  Write interface to pybind11, time-consuming a.s.

For 1, speed is not a concern since the training is done offline anyways. It takes 2 hours to compute 5 million datapoints (Heston) and store them in MySQL for later processing.

Re: Python tricks

Posted: July 19th, 2020, 12:43 pm
by Cuchulainn
I'll post the code some time soon. It will be Open Source.
How's it going here? we would like to test it.
Precise, step-by-step documentation is essential.