py310-cython @0.29.32: error: invalid command 'egg_info'

[ronf (Ron Frederick)]

I recently upgraded my Mac to Ventura and have not been able to install py310-cython (which a number of other ports depend on). The problem appears to be related to setuptools/distutils, but attempts to uninstall and reinstall those haven't made a difference. Here's some of the relevant log output from the failure:

:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'python_requires'
:info:build   warnings.warn(msg)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'zip_safe'
:info:build   warnings.warn(msg)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'entry_points'
:info:build   warnings.warn(msg)
:info:build usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
:info:build    or: setup.py --help [cmd1 cmd2 ...]
:info:build    or: setup.py --help-commands
:info:build    or: setup.py cmd --help
:info:build error: invalid command 'egg_info'
:info:build ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-cython/py310-cython/work/Cython-0.29.32" && /opt/local/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m build --wheel --no-isolation --outdir /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-cython/py310-cython/work
:info:build Exit code: 1

This is on a fresh install of MacPorts and all its pots after the OS upgrade, following the usual "Migration" documentation. Prior to the upgrade, the python install was working fine.

[ronf (Ron Frederick)]

Full log from attempt to install py310-cython

[kencu (Ken)]

I can't reproduce this - py310-cython installs without trouble for me:

% port -v installed py310-cython
The following ports are currently installed:
  py310-cython @0.29.32_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T21:42:35-0800'

I installed it with no ports active, and let it call in what it wanted:

% port -v installed | grep active
  bzip2 @1.0.8_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:30:37-0800'
  cython_select @0.1_2 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:11-0800'
  expat @2.5.0_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:35:24-0800'
  gettext-runtime @0.21_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:19:55-0800'
  libedit @20221030-3.1_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:35:33-0800'
  libffi @3.4.4_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:25:58-0800'
  libiconv @1.17_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:19:19-0800'
  ncurses @6.3_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:20:28-0800'
  openssl @3_8 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:36:36-0800'
  openssl3 @3.0.7_2+legacy (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:36:31-0800'
  py310-build @0.9.0_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:11-0800'
  py310-cython @0.29.32_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T21:42:35-0800'
  py310-installer @0.6.0_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:12-0800'
  py310-packaging @22.0_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:11-0800'
  py310-pep517 @0.13.0_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:11-0800'
  py310-setuptools @65.6.3_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:13-0800'
  py310-tomli @2.0.1_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:11-0800'
  py310-wheel @0.38.4_0 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T21:42:13-0800'
  python3_select @0.0_3 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T19:36:38-0800'
  python310 @3.10.9_0+lto+optimizations (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:39:42-0800'
  python_select @0.3_10 (active) requested_variants='' platform='darwin 22' archs='noarch' date='2022-12-25T19:36:38-0800'
  sqlite3 @3.40.0_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:37:01-0800'
  xz @5.2.9_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:27:39-0800'
  zlib @1.2.13_0 (active) requested_variants='' platform='darwin 22' archs='arm64' date='2022-12-25T19:27:43-0800'

[jmroot (Joshua Root)]

The most recent error in the log is different to the one quoted in the ticket description:

:info:build ModuleNotFoundError: No module named 'setuptools.build_meta'

What does port installed py310-setuptools say? What happens if you start MacPorts python3.10 in your terminal and run import setuptools.build_meta?

[ronf (Ron Frederick)]

Thanks for looking into this!

I didn't have a second ARM64 machine available here to test on, but I was able to spin up a fresh macOS Ventura VM and install Xcode and MacPorts on it, and I can confirm in a completely fresh install I also did not see the problem. The py310-cython package and all its dependencies installed just fine. I'm going to take a closer look at the resulting files on the two systems, especially around files from setuptools, to see what differences might exist.

What's odd here is that going through the "Migration" steps after upgrading to Ventura involved uninstalling ALL of the ports and installing fresh copies of all of them into an empty repository. So, other than the order ports were installed, it should have been very close to this fresh install. Also, manually uninstalling and reinstalling things like py310-setuptools hasn't helped.

Here's what I get for "port installed py310-setuptools":

The following ports are currently installed:
  py310-setuptools @65.6.3_0 (active)

There doesn't appear to be any "build_meta" in setuptools, though:

ultra:~#python3.10
Python 3.10.9 (main, Dec 28 2022, 13:05:08) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools.build_meta
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools.build_meta'

[ronf (Ron Frederick)]

Checking the fresh VM, "import setuptools.build_meta" does work there, so perhaps there's some older version of setuptools which is being picked up on the machine with the error, despite the fact that both claim to have version 65.6.3_0 installed.

[ronf (Ron Frederick)]

Ok - I checked the setuptools directory, and definitely found some weirdness. For instance, init.py didn't exist on my system even after a setuptools reinstall, but there was a file called init.py..mp_1648743825. That generally happens when multiple packages have found conflicts in which port owns a given file. Even after wiping out everything in /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools and doing a forced uninstall/reinstall, I'm still getting errors, though:

I don't see anything about build_meta after manually cleaning out the setuptools site-packages dir. After that, "import setuptools.build_meta" worked on the original system, but the build is still failing, with the error about egg_info and some of the config options not being recognized:

:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'python_requires'
:info:build   warnings.warn(msg)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'zip_safe'
:info:build   warnings.warn(msg)
:info:build /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'entry_points'
:info:build   warnings.warn(msg)
:info:build usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
:info:build    or: setup.py --help [cmd1 cmd2 ...]
:info:build    or: setup.py --help-commands
:info:build    or: setup.py cmd --help
:info:build error: invalid command 'egg_info'
:info:build ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel
:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-cython/py310-cython/work/Cython-0.29.32" && /opt/local/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m build --wheel --no-isolation --outdir /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_python_py-cython/py310-cython/work
:info:build Exit code: 1

[kencu (Ken)]

when this happens, often the best solution is to nuke your installation of MacPorts and reinstall your ports….

[ronf (Ron Frederick)]

Sadly, after spending a couple of hours attempting to reinstall all the ports again from scratch, I'm seeing the same error. Everything else installs just fine, but py310-cython is still complaining about egg_info and the various other config options above. So, there's something outside of whatever MacPorts is writing to that's causing this, but I have no idea why it isn't affecting any of the other ports.

[ronf (Ron Frederick)]

Ok - I think I found it. Somehow, in addition to setuptools-65.6.3.dist-info being present in /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages, there was also a directory named setuptools-62.1.0.dist-info present alongside it. Even though the "setuptools" in that same parent directory appeared to be the new version, somehow the presence of this older version was confusing things. Once that was removed, the cython install succeeded. Once again, I have no idea why cython was sensitive to this, though, while nothing else using setuptools in python 3.10 had any problems.

Sorry for wasting your time here. I appreciate the help!

[kencu (Ken)]

glad you got it sorted out

[jmroot (Joshua Root)]

Sounds like you may have used pip or similar to install modules into the MacPorts prefix at some point. That's not recommended because it can cause just these kind of conflicts. Best to use pip's --user option or install into a venv.

[ronf (Ron Frederick)]

Yeah, thanks - I need to start using venv more. What I still don't understand is why cython is the only port that gave me a problem here, though.