VTK +python[version] does not provide `.dist-info`

[jjstickel (Jonathan Stickel)]

I'm working on updating the py-mayavi port and found that the latest version complains that vtk's python wrappers are not installed, even though I have installed vtk +python311. I think it is looking for vtk-[version].dist-info (or vtkmodules-[version].dist-info) in .../site-packages/. The python[version] variants should generate and install an appropriate .dist-info folder.

[Dave-Allured (Dave Allured)]

@jjstickel If you know how to enable this in the vtk portfile, please submit a PR, or offer more advice. Thanks.

[jjstickel (Jonathan Stickel)]

Replying to Dave-Allured:

@jjstickel If you know how to enable this in the vtk portfile, please submit a PR, or offer more advice. Thanks.

Yes, of course. If it was obvious to me, I would have done it already. I see that vtk is in PyPi, which I am sure provides .dist-info. I'll see if I can find some time next week to reverse-engineer how it is done there.

[Dave-Allured (Dave Allured)]

Not being a pythoner, I do not know anything about dist-info's. Is it a standard python device? Is it already somewhere in the VTK source, maybe hidden?

I suggest ask on the VTK user forums how to do this. A year or two back, they were responsive to a naive question from me. Perhaps .dist-info is already implemented, and it is simply a matter of enabling or debugging. Also I would check their gitlab repo.

[Dave-Allured (Dave Allured)]

Also I see that VTK is very close to releasing 9.4.0. Probably check that to see if they are adding .dist-info.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jjstickel:

py-mayavi […] complains that vtk's python wrappers are not installed

What's the exact error message, so we can search for it?

Replying to jjstickel:

I see that vtk is in PyPi, which I am sure provides .dist-info.

If what vtk's python variants provides can instead be provided by separate py-vtk ports installed from pypi, that would usually be preferable for the usual reasons of not needing to recompile vtk just to get different python bindings and other ports being able to declare dependencies on the python bindings they need.

[jmroot (Joshua Root)]

Replying to Dave-Allured:

Not being a pythoner, I do not know anything about dist-info's. Is it a standard python device?

​https://packaging.python.org/en/latest/specifications/recording-installed-packages/

[Dave-Allured (Dave Allured)]

Replying to jmroot:

​https://packaging.python.org/en/latest/specifications/recording-installed-packages/

So is it reasonable to ask upstream to add this python project info into their VTK package? Or is this an expected function of portfiles?

[jjstickel (Jonathan Stickel)]

Replying to ryandesign:

Replying to jjstickel:

py-mayavi […] complains that vtk's python wrappers are not installed

What's the exact error message, so we can search for it?

ERROR Missing dependencies:
	vtk

That's it. I'm not a python-packaging expert, but I think some setuptools protocol is checking for the vtk dependency and not finding it because there is no .dist-info folder associated with the vtk port.

[jmroot (Joshua Root)]

Replying to Dave-Allured:

So is it reasonable to ask upstream to add this python project info into their VTK package? Or is this an expected function of portfiles?

It's something that needs to happen to make the python package discoverable by other python code, so the upstream build system installing a dist-info along with the python bindings would be ideal. But I guess if it doesn't, the port should do it. You can look at the py-oldest-supported-numpy port for an example of installing a very minimal dist-info.

[jjstickel (Jonathan Stickel)]

I tested the install of vtk via pip. It does install a binary wheel (vtk-9.3.1-cp311-cp311-macosx_11_0_arm64.whl) and includes the .dist-info folder. I am not so familiar how wheels are built, but I'll try to find the source for vtk.

[jjstickel (Jonathan Stickel)]

I found Documentation/docs/advanced/build_python_wheels.md in the VTK sources, which indicates that the variable VTK_WHEEL_BUILD can be used to create a setup.py file for the purpose of building the wheel. I'm still not sure how the right way to install both the non-python vtk libraries and the contents of the wheel in a way that is not duplicative. I'll continue working towards a solution and a PR, unless someone else gets to it first.

Replying to Dave Allured's suggestion to inquire with VTK developers, I will indeed do so if needed and I know the right question to ask.

[jjstickel (Jonathan Stickel)]

I now know that a standard build, even with python wrappers, is incompatible with a wheels build. I discovered this empirically before I found this report:

​https://gitlab.kitware.com/vtk/vtk/-/issues/18685

I did ask on vtk's discourse, but I doubt there will be an easy resolution:

​https://discourse.vtk.org/t/create-python-dist-info-folder-without-wheel-build/14814

So, it doesn't look there is a good option. What comes to my mind is to strip the python variants out of the vtk port and instead have them reside in a separate port (py-vtk) built via the wheels infrastructure. From what I can tell, this would duplicate the c++ vtk libraries, one set in /opt/local/lib via the vtk port, and another set in /opt/local/Library/Frameworks/Python.framework/... via py-vtk. py-vtk need not depend on vtk, and they should not conflict. Another thought is to just drop py-mayavi from macports entirely since it can now be installed reliably via pip. What do others think?

[Dave-Allured (Dave Allured)]

Replying to jjstickel:

Thank you for all that research. Ideally the vtk port should build the c++ vtk libraries as Macports standard in /opt/local/lib, and they should not be duplicated. Can the VTK build system be coerced into building a separate port as python wrappers for the core c++ libraries in /opt/local/lib, without too much trouble? Speaking naively, it seems that sort of thing has been done elsewhere, many times.

[Dave-Allured (Dave Allured)]

Can the VTK build system be coerced into building a separate port as python wrappers for the core c++ libraries in /opt/local/lib, without too much trouble?

Uh, let me rephrase that. Can the VTK build system be coerced to specifically build its python wrappers around external, pre-built, non-wheels VTK c++ libraries? Start by assuming that the core and wrappers are the same version.

[jjstickel (Jonathan Stickel)]

Replying to Dave-Allured:

I don't think so, at least not by the documentation and intended use cases that I'm aware of. You're welcome to ask on the discourse post I started.

I agree with your preference to use the same base vtk libraries. I'm noticing a trend in open-source software where it is common practice to duplicate dependencies for each minutely different situation, and Python is a leading offender by promoting virtual environments. So bundling a separate set of vtk libraries in the vtk-python wheel build is simply going along with that trend. Oh, and Paraview is another situation where it builds and uses its own set of vtk libraries. At least, the last time I looked into building paraview, using already existing vtk libraries was "in development". So I'm not sure pythoners and vtkers will even understand what we are asking.

[jjstickel (Jonathan Stickel)]

I think I have a solution worked out in ​https://github.com/macports/macports-ports/pull/26442 . It's a bit of hack, where cmake configure is run again in post-destroot in order to generate setup.py. This is then used to create the .dist-info files. Although we could create the files "from scratch", it would be a bit troublesome to replicate METADATA precisely. Let me know what you think in the PR comments.