py38-dns-lexicon @3.4.3_0 certbot execution dns-lexicon missing

[ArtKoKo]

certbot @1.8.0_0+python38 (active)
certbot-dns-ovh @1.8.0_0+python38 (active)
py38-dns-lexicon @3.4.3_0 (active)

simple execution : sudo certbot

An unexpected error occurred: pkg_resources.DistributionNotFound: The 'dns-lexicon>=2.7.14' distribution was not found and is required by the application

[ArtKoKo]

log of certbot

[mf2k (Frank Schima)]

In the future, please add the port maintainer(s) to Cc (port info --maintainers certbot-dns-ovh), if any.

[ArtKoKo]

Replying to mf2k:

In the future, please add the port maintainer(s) to Cc (port info --maintainers certbot-dns-ovh), if any.

Thanks. Noted

[Schamschula (Marius Schamschula)]

Hmm. certbot itself does not require dns-lexicon.

However, as I have certbot-dns-namecheap, and thus py-dns-lexicon, installed I see the same error.

The problem is with py-dns-lexicon, as it currently doesn't install any useful files.

Looking into it.

[Schamschula (Marius Schamschula)]

As of the 3.4.0 release dns-lexicon uses poetry. I'm not familiar with poetry.

[ArtKoKo]

Replying to Schamschula:

Hmm. certbot itself does not require dns-lexicon.

However, as I have certbot-dns-namecheap, and thus py-dns-lexicon, installed I see the same error.

The problem is with py-dns-lexicon, as it currently doesn't install any useful files.

Looking into it.

Indeed all the ports certbot-dns-* depend on py-dns-lexicon.
a possible improvement ?

[ArtKoKo]

I'm not familiar myself with poetry but it's yet another package manager to facilitate dependency management : ​https://python-poetry.org/docs

dgilman can maybe help us? ​https://ports.macports.org/maintainer/github/dgilman/

[Schamschula (Marius Schamschula)]

No, certbot does not need certbot-dns-namecheap to be installed. When I deactivated certbot-dns-namecheap and then reinstalled certbot, certbot worked correctly. However, this will loose functionality for some users.

I think the best route is to back py-dns-lexicon up to the last pre-poetry version, i.e. 3.3.28.

[ArtKoKo]

Yescerbot does not need py-dns-lexicon but cerbot-dns-* does. I agree to come back to the previous version but how ?

[Schamschula (Marius Schamschula)]

In 981f414eeb62fcef482a322444f942e87dcbdf86/macports-ports (master):

py-dns-lexicon: revert to 3.3.28

we don't want to use poetry to build
remove github PG, let python PG do the work
See: #61190

[ArtKoKo]

For info, the response of the author of lexicon :

In fact you can continue to use your workflow with pip/setuptools as before, and not care about poetry if you want, thanks to two mechanisms available here:
recent versions of pip (>=10.0), implements PEP 517 and PEP 518, which is a convention to prepare in isolation the build system required in the poetry.toml of a project. Basically you can checkout the project, run pip install lexicon, and pip will fetch poetry on its own to install the project. There is a limitation for editable installations (pip install -e) which fall in the next point for older versions of pip, or for editable installations, an installation through setuptools is required and still possible. Indeed a shim for setup.py has been designed by the packaging community for retro-compatibility purpose, and I added it to Lexicon. In this case setuptools will kick in and you will not benefit from Poetry features, like advanced dependency graph resolution, but for your case I think it will be sufficient.

[Schamschula (Marius Schamschula)]

We don't use pip in our installation process. Just like poetry, pip manages python projects and their dependencies in parallel to MacPorts. This is a bad idea.

Using setuptools, nothing gets installed, other that a few stub files:

port contents py38-dns-lexicon
Port py38-dns-lexicon contains:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/PKG-INFO
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/SOURCES.txt
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/dependency_links.txt
  /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8.egg-info/top_level.txt

which, for the most part are empty.

[ArtKoKo]

Response of AnalogJ/lexicon

No it is me. The shim does not cover the case of old versions of pip, just the case of pip install -e (editable mode). You still need at least pip 10.0+ to make it work through the build isolation. Having a more recent version of pip on FreeBSD is not doable ?

[Schamschula (Marius Schamschula)]

macOS != FreeBSD. I run FreeBSD on my servers. Fresh Ports != MacPorts. There are major differences how these two port systems work.

Yes, on MacPorts, we have a current version of pip:

port list py38-pip
py38-pip                       @20.2.3         python/py-pip

and poetry:

port list poetry
poetry                         @1.0.3          python/poetry

However, any package installed by pip may conflict with the same package installed by MacPorts. I.e. if I install a package via pip it may install a number dependencies met previously installed by MacPorts. These packages are not recorded in the MacPorts registry. If you then attempt to install the corresponding MacPorts package, the activation phase will fail, as the files will already exist. This is the same issue as with poetry.

For example: before there was a port of tensorflow, I used pip to install tensorflow. Later, when I used MacPorts to install tensorflow, I had to manually uninstall several packages that had been installed by pip.

I go out of my way not to depend on pip to avoid package management issues.

[ArtKoKo]

Response of AnalogJ/lexicon:

Also, depending of the use case:

if it is for development, you should anyway use a Python virtual environment, and in it you will be able to use an up-to-date version of pip
if it is to use it, the wheel is better than installing from sources ​https://files.pythonhosted.org/packages/4b/3b/dcaa4f6a0930703c73d44e3c6c7520132a353a640180b9da1b5b3436ccb6/dns_lexicon-3.4.3-py3-none-any.whl

[ArtKoKo]

It's a long shot...

[Schamschula (Marius Schamschula)]

This port is not for development: py-dns-lexicon is a dependency of certbot-dns-namecheap which is used for production instances of certbot.

MacPorts users want their certbot certificates to be automatically maintained.

[ArtKoKo]

Hopefully in the future the modules certbot-dns-* will use something other than py-dns-lexicon because the developer is taking a wrong path for us...

[ArtKoKo]

Question at AnalogJ/lexicon:

I spoke with one of the maintainers of the package manager MacPorts:
The problem is that lexicon is a dependency of the certbot-dns extensions of certbot and that adding another package manager to handle the dependencies of the dependency obviously becomes counterproductive...

Response of AnalogJ/lexicon:

Of course, but MacPorts maintainers need to find a solution for packages moving to PEP 571, because more and more packages will move to pyproject.toml since it is a standard now.
I looked at MacPorts behavior, but failed to understand how a Python package is actually installed. I suppose it is something like python setup.py install, which is exclusive to setuptools. For a maximum compatibility I would recommend to use pip: in fact, pip would not be used as package manager here, but as the build frontend to install the package, with a command like pip install --no-deps. In this case pip will leverage the actual installation to setuptools for legacy, and the defined build system for PEP 517 compliant projects.

[ArtKoKo]

Going back to the old version seems to bring another missing port:
pkg_resources.DistributionNotFound: The 'dnspython<2requests[security]' distribution was not found and is required by dns-lexicon

murphy's law ?

[Schamschula (Marius Schamschula)]

Nope.

Just a package that I no longer use myself. I created the certbot port before all the plug-ins were added. In the meantime, I moved both my external servers to FreeBSD.

I keep updating the packages with new releases, but only do very basic functionality tests, as my build machine only runs an intranet web server.

As to the error, I didn't see it myself, when invoking certbot.

A quick search yields ​https://bugs.archlinux.org/task/67480

so I need to add py-dnspython to the dependencies of py-dns-lexicon.

[ArtKoKo]

It's fine.
I add py-dnspython and it works very well \\ Thank you Schamschula !

[Schamschula (Marius Schamschula)]

In 60c1ef5b5a511d8b960e2ad4b576ae33cf77e607/macports-ports (master):

py-dns-lexicon: add py-dnspython dependency

See: #61190

[ArtKoKo]

Hi Schamschula,

for info, a new dependency appear py-filelock :

pkg_resources.DistributionNotFound: The 'filelock>=3.0.8' distribution was not found and is required by tldextract

Works well after intallation...

[Schamschula (Marius Schamschula)]

In 0f2c198a531b60807f76932ecad31a4575b7b2b1/macports-ports (master):

py-tldextract: add py-filelock dependancy

See: #61190#comment:24

[ryandesign (Ryan Carsten Schmidt)]

py-dns-lexicon was updated to 3.5.2 and uses the new python.pep517 yes functionality that was added to the python 1.0 portgroup. Does that resolve this issue?

[Schamschula (Marius Schamschula)]

See: ​https://github.com/macports/macports-ports/commit/38b820ff480eb310b89557e3ecdd0b63b552f03a

[ArtKoKo]

Yes I have no trouble with the last lexicon version, well done !