Some Python distfiles not updated

[erikbs]

I am trying to update a few packages for Python 3.12, but some of them (or, at least one) require a newer version to work with Python 3.12. So I went ahead and tried to update e.g. py-lxml from 4.9.1 to 4.9.3, as its release notes mention Python 3.12 explicitly (4.9.3 came out in July). Unfortunately MacPorts then complained that lxml-4.9.3.tar.gz could not be downloaded, and sure enough there is no 4.9.3 in ​https://distfiles.macports.org/py-lxml/. However, I do not see 4.9.2 either, even though it came out nearly a year ago. On the other hand, other Python packages (such as py-fonttools) seem to be mirrored much more often.

Question is: are Python distfiles mirrored automatically?

If no:

• Is is possible to trigger a sync for a particular package?
• How do I do it?

If yes:

• Why has not py-lxml been mirrored in a year?
• Is this a bug?

[jmroot (Joshua Root)]

Distfiles are mirrored for a port when it is built on the buildbot, which happens when an update is pushed to the master branch of macports-ports. It's expected that there will be no distfiles mirrored for versions that don't exist in the official ports tree. That doesn't prevent fetching the distfiles from the master_sites specified in the Portfile (which may use defaults specified in e.g. the python portgroup). In fact, if fetching is failing locally, it's likely that mirroring will also fail, because it uses the same mechanism.

[ryandesign (Ryan Carsten Schmidt)]

No bug here.

[erikbs]

Replying to jmroot:

Distfiles are mirrored for a port when it is built on the buildbot, which happens when an update is pushed to the master branch of macports-ports. It's expected that there will be no distfiles mirrored for versions that don't exist in the official ports tree. That doesn't prevent fetching the distfiles from the master_sites specified in the Portfile (which may use defaults specified in e.g. the python portgroup). In fact, if fetching is failing locally, it's likely that mirroring will also fail, because it uses the same mechanism.

Makes sense! Unfortunately it looks like MacPorts does not try to fetch distfiles from PyPI, even though pypi is specified in the master_sites section of the portgroup file. I double checked that version 4.9.3 is in fact present on PyPI, so there must be something that is not working properly in my setup.

Given that there is no bug here, I guess I will have to create a pull request for updating to 4.9.3 and let the CI check if it works, but I would have preferred being able to test things locally first.

[jmroot (Joshua Root)]

Replying to erikbs:

Unfortunately it looks like MacPorts does not try to fetch distfiles from PyPI, even though pypi is specified in the master_sites section of the portgroup file.

Now that would indeed be a bug, though I strongly suspect it might be caused by something in your Portfile, given that nobody else has reported this behaviour. Mostly likely it will all get sorted out in the course of reviewing the PR.

[erikbs]

It passed CI with the exact same Portfile that I tried locally, so there must be something that caused fetching to fail on my computer. I do run an older OS X version (10.9), but I would have expected at least an entry in the log saying that it had tried contacting pypi.org?

I managed to test locally just now by manually putting the new version in /opt/local/var/macports/distfiles/py-lxml:

[/] upgrade py311-lxml
--->  Computing dependencies for py311-lxml
--->  Fetching archive for py311-lxml
--->  Attempting to fetch py311-lxml-4.9.3_0.darwin_13.x86_64.tbz2 from http://packages.macports.org/py311-lxml
--->  Attempting to fetch py311-lxml-4.9.3_0.darwin_13.x86_64.tbz2 from http://cph.dk.packages.macports.org/py311-lxml
--->  Attempting to fetch py311-lxml-4.9.3_0.darwin_13.x86_64.tbz2 from http://fra.de.packages.macports.org/py311-lxml
--->  Fetching distfiles for py311-lxml
--->  Verifying checksums for py311-lxml
--->  Extracting py311-lxml
--->  Applying patches to py311-lxml
--->  Configuring py311-lxml
--->  Building py311-lxml
--->  Staging py311-lxml into destroot
--->  Installing py311-lxml @4.9.3_0
--->  Cleaning py311-lxml
--->  Computing dependencies for py311-lxml
--->  Deactivating py311-lxml @4.9.1_0
--->  Cleaning py311-lxml
--->  Activating py311-lxml @4.9.3_0
--->  Cleaning py311-lxml
--->  Scanning binaries for linking errors
--->  No broken files found.
--->  No broken ports found.

[erikbs]

Pull request: ​https://github.com/macports/macports-ports/pull/21318

[kencu (Ken)]

pulling unmirrored distfiles from python sources is especially sensitive to older curl versions that macports may be built against.

I currently have to build macports sgainst a newer libcurl on all systems up to 10.14 for full functionality.

see long discussion with workarounds here:

#51516

[erikbs]

Replying to kencu:

pulling unmirrored distfiles from python sources is especially sensitive to older curl versions that macports may be built against.

I currently have to build macports sgainst a newer libcurl on all systems up to 10.14 for full functionality.

see long discussion with workarounds here:

#51516

Good idea! Building the bootstrap version of MacPorts took only a few minutes, but I run into a dependency cycle when attempting to install curl. I cannot build xz, because its (indirect) dependency gettext-tools-libs (through gettext) requires a newer Clang version (e.g., 3.7) to compile successfully, but Clang 3.7 needs xz for extraction …

Both zlib and llvm-3.7 replace xz with xz-bootstrap on Darwin < 11. LLVM 3.7 builds successfully by adjusting that to Darwin < 14 (from what I read, Yosemite does not have /usr/bin/xz either, but Yosemite already gets a Clang version based on 3.7 through Xcode, whereas Mavericks is stuck on one based on 3.5).

Clang 3.7 ran into another dependency cycle with Perl 5.34, but removing the analyzer variant got rid of the Perl trouble. It did not help much, as a immediately ran into another cycle with libxml2 because of its dependency on xz. I got around that by taking the xz-bootstrap lines from llvm-3.7 (see ​https://github.com/erikbs/macports-ports/commit/5fc8e53971d9cc88d4f736fa44ea751b9ccc4832).

None of this was necessary when I last reinstalled 10.9, but then that is five years ago and things could have changed.

I am considering making a pull request, but I am unsure if I should touch libxml2. According to the llvm-3.7 Portfile, Clang does not really depend on libxml2 (“c-index-test will optionally use it if it is installed”), so on Darwin < 11 the whole dependency is removed.

I did roughly this (I had an unrelated kernel panic in the process and lost my notes):

# Build bootstrap
curl -ROL 'https://github.com/macports/macports-base/releases/download/v2.8.1/MacPorts-2.8.1.tar.bz2'
tar xf MacPorts-2.8.1.tar.bz2
cd MacPorts-2.8.1
./configure --prefix=/opt/bootstrap --with-curlprefix=/usr
sudo make install
make distclean

# Install SSL/curl
sudo /opt/bootstrap/bin/port
selfupdate
# port edit <llvm-3.7 | libxml2 | zlib> and make the edits mentioned above
install --unrequested clang-3.7 -analyzer
install curl
setrequested readline
reclaim
exit

# Remove things that just take up lots of space
sudo rm -rf /opt/bootstrap/var/macports/sources
sudo rm -rf /opt/bootstrap/var/macports/distfiles
sudo rm -rf /opt/bootstrap/share/man
sudo rm -rf /opt/bootstrap/libexec/openssl3/share/man
sudo rm -rf /opt/bootstrap/share/doc

# Rebuild MacPorts
./configure --with-curlprefix=/opt/bootstrap --enable-readline
make

… and that is where I am now. readline.c fails to compile. Readline support is more important than not having to copy distfiles in manually, so I am not sure if I bother going further with this now.

If we cannot have comment:ticket:51516:99, I think your "minimalcurl" idea is a good solution too.

[jmroot (Joshua Root)]

Replying to erikbs:

It passed CI with the exact same Portfile that I tried locally, so there must be something that caused fetching to fail on my computer. I do run an older OS X version (10.9), but I would have expected at least an entry in the log saying that it had tried contacting pypi.org?

OK, it was the claim that PyPI wasn't even tried that was throwing me off, but it's been made clear in the PR that it is tried and fails.

[erikbs]

Yes, I was so focused on finding pypi.org/pypi/PyPI/pip in the log that I completely missed pythonhosted.org (I was not aware of that domain). Thanks for the help! Without another pair of eyes on it I would probably have been still staring at this log

[kencu (Ken)]

I don’t seem to ever enable readline support… everything seems fine to me.

perhaps I don’t know what I’ve been missing all these years.