Clever Geek Handbook

How can I create a Python wheel with a compiled Fortran extension without requiring a specific MinGW version on the user's system?

As far as I understand, one of the main benefits of distributing Python packages via wheels is that I can include extension modules in the compiled form. Then the user of the package does not need to have a system that allows compiling the source code.

I have now managed to create a wheel for my package, which includes the Fortran plugin. The computer I built on has Windows7 64 and Python 3.6.

To get things up and running, I followed this very helpful recommendation (many thanks to Michael Hirsch). One of the steps was to install MinGW-64 with the following settings: Architecture: x86_64, Themes: posix, Exception: seh.

Then I installed the Python package on another test machine (Win10 64, Python 3.6) from this wheel:

D:\dist2>pip install SMUTHI-0.2.0a0-cp36-cp36m-win_amd64.whl
Processing d:\dist2\smuthi-0.2.0a0-cp36-cp36m-win_amd64.whl
Requirement already satisfied: scipy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: sympy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: argparse in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: numpy in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: matplotlib in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: pyyaml in c:\programdata\anaconda3\lib\site-packages (from SMUTHI==0.2.0a0)
Requirement already satisfied: six>=1.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: python-dateutil in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pytz in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: cycler>=0.10 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in c:\programdata\anaconda3\lib\site-packages (from matplotlib->SMUTHI==0.2.0a0)
Installing collected packages: SMUTHI
Successfully installed SMUTHI-0.2.0a0

However, when I started a test run of the program, I encountered the following error:

D:\dist2>smuthi example_input.dat
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\smuthi.exe\__main__.py", line 5, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\__main__.py", line 4, in <module>
    import smuthi.read_input
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\read_input.py", line 3, in <module>
    import smuthi.simulation
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\simulation.py", line 8, in <module>
    import smuthi.t_matrix as tmt
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\t_matrix.py", line 6, in <module>
    import smuthi.nfmds.t_matrix_axsym as nftaxs
  File "c:\programdata\anaconda3\lib\site-packages\smuthi\nfmds\t_matrix_axsym.py", line 11, in <module>
    import smuthi.nfmds.taxsym
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

The extension file .pyd

( taxsym.cp36-win_amd64.pyd

) was in place - it was just that Python couldn't load it.

Then I uninstalled MinGW from the test machine and reinstalled MinGW-64 with the same settings I used on the construction machine (see above). Afterwards, I could run the program and Python was able to load the plugin correctly.

My question is, does anyone know why the error occurred in the first place? And how can I prevent the user of my Python package from having a specific version of MinGW (or even any) installed for the package to work?

Edit: a small example that reproduces the error:

Minimal example

File structure:

setup.py
example/
    __init__.py
    run_hello.py
    extension_package/
        __init__.py             
        fortran_hello.f90

setup.py

is reading:

import setuptools
from numpy.distutils.core import Extension
from numpy.distutils.core import setup

setup(
   name="example",
   version="0.1",
   author="My Name",
   author_email="my@email.com",
   description="Example package to demonstrate wheel issue",
   packages=['example', 'example.extension_package'],
   ext_modules=[Extension('example.extension_package.fortran_hello',
                          ['example/extension_package/fortran_hello.f90'])],
)

run_hello.py

is reading:

import example.extension_package.fortran_hello
example.extension_package.fortran_hello.hello()

fortran_hello.f90

is reading:

subroutine hello
print *,"Hello World!"
end subroutine hello

Wheel creation

I ran python setup.py bdist_wheel

, which resulted in the fileexample-0.1-cp36-cp36m-win_amd64.whl

Installing the package on a machine with the correct MinGW version

D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1

D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
 Hello World!
>>> exit()

This is how it should be.

Installing a package on a machine without the correct MinGW version

To reproduce the error, I renamed the MinGW folder on the test machine to a different name and then:

D:\dist>pip install example-0.1-cp36-cp36m-win_amd64.whl
Processing d:\dist\example-0.1-cp36-cp36m-win_amd64.whl
Installing collected packages: example
Successfully installed example-0.1

D:\dist>python
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import example.run_hello
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\site-packages\example\run_hello.py", line 1, in <module>
    import example.extension_package.fortran_hello
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
+3


source to share


1 answer


I recently ran into this problem by writing my own f2py toolchain, chaining and linking all the components separately. The script found or installed the required compilers automatically if not already found in the path. In cases where the gfortran tools depended on a path but were present on the machine, I was able to inject the correct environment variables in os.environ

and invoke compiler calls with Popen

and a set of environment variables to get pyd to compile. But outside of this python instance the environment variables were not correct to run pyd, I was getting the same error DLL load failed

even on the same machine that pyds was compiled on, but which did not have the correct path setting.

So, since I am compiling all the steps separately, only using f2py to generate the f and c frames, I just added -static -static-libgfortran -static-libgcc

links to my step and this forces pyd to include the required libraries to run on those machines without the correct environment variables.

Achieving the same use of numpy.distutils is possible (thanks to https://github.com/numpy/numpy/issues/3405 ):

from numpy.distutils.core import Extension, setup


if __name__ == "__main__":
    setup(
        name="this",
        ext_modules=[
            Extension("fortmod_nostatic",
                      ["src/code.f90"],
                      ),
            Extension("fortmod_withstatic",
                      ["src/code.f90"],
                      extra_link_args=["-static", "-static-libgfortran", "-static-libgcc"]
                      )
        ]
    )


I put the above in a file test.py

and built withpython test.py build_ext --inplace --compiler=mingw32 --fcompiler=gnu95 -f

By comparison, there is a clear difference . Inspecting pyd with a walking dependency reveals nostatic which depends on libgfortran-4.dll

, while additional flags generate pyd that doesn't depend on that library . In my case, after adding static flags, a machine without the correct environment variables can start pyds, and I suspect this case will be similar to yours since the libgfortran dependency has been removed.

Hope it helps! my first SO post ..

+2


source






More articles:


All Articles