Trace mode does not fully hide python modules

[ryandesign (Ryan Carsten Schmidt)]

py310-libxml2 and later build fine but py39-libxml2 and earlier are failing to build:

DEBUG: system:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-libxml2/py39-libxml2/work/libxml2-2.12.5/python" && /opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 setup.py --no-user-cfg build -j8 
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-libxml2/py39-libxml2/work/libxml2-2.12.5/python/setup.py", line 8, in <module>
    from setuptools import setup, Extension
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 14, in <module>
    from . import version as _version_module
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/version.py", line 1, in <module>
    from ._importlib import metadata
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_importlib.py", line 43, in <module>
    disable_importlib_metadata_finder(metadata)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_importlib.py", line 31, in disable_importlib_metadata_finder
    to_remove = [
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/setuptools/_importlib.py", line 34, in <listcomp>
    if isinstance(ob, importlib_metadata.MetadataPathFinder)
AttributeError: module 'importlib_metadata' has no attribute 'MetadataPathFinder'

All that py-libxml2 is trying to is use its setup.py file. This importlib_metadata requirement is not mentioned anywhere in libxml2, so I think this is a py-setuptools bug?

[ryandesign (Ryan Carsten Schmidt)]

Since this used to work, I wanted to try downgrading py-setuptools to 69.0.3. It doesn't build; it has the same error:

:debug:build system:  cd "/opt/local/var/macports/build/_private_tmp/py39-setuptools/work/setuptools-69.0.3" && /opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -m build --no-isolation --wheel --outdir /opt/local/var/macports/build/_private_tmp/py39-setuptools/work --skip-dependency-check 
:info:build * Building wheel...
:info:build Traceback (most recent call last):
:info:build   File "/opt/local/share/py-bootstrap-modules/pep517/in_process/_in_process.py", line 351, in <module>
:info:build     main()
:info:build   File "/opt/local/share/py-bootstrap-modules/pep517/in_process/_in_process.py", line 333, in main
:info:build     json_out['return_val'] = hook(**hook_input['kwargs'])
:info:build   File "/opt/local/share/py-bootstrap-modules/pep517/in_process/_in_process.py", line 249, in build_wheel
:info:build     return _build_backend().build_wheel(wheel_directory, config_settings,
:info:build   File "/opt/local/share/py-bootstrap-modules/pep517/in_process/_in_process.py", line 77, in _build_backend
:info:build     obj = import_module(mod_path)
:info:build   File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
:info:build     return _bootstrap._gcd_import(name[level:], package, level)
:info:build   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
:info:build   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
:info:build   File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
:info:build   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
:info:build   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
:info:build   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
:info:build   File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
:info:build   File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
:info:build   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
:info:build   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
:info:build   File "/opt/local/var/macports/build/_private_tmp/py39-setuptools/work/setuptools-69.0.3/setuptools/__init__.py", line 13, in <module>
:info:build     from . import version as _version_module
:info:build   File "/opt/local/var/macports/build/_private_tmp/py39-setuptools/work/setuptools-69.0.3/setuptools/version.py", line 1, in <module>
:info:build     from ._importlib import metadata
:info:build   File "/opt/local/var/macports/build/_private_tmp/py39-setuptools/work/setuptools-69.0.3/setuptools/_importlib.py", line 43, in <module>
:info:build     disable_importlib_metadata_finder(metadata)
:info:build   File "/opt/local/var/macports/build/_private_tmp/py39-setuptools/work/setuptools-69.0.3/setuptools/_importlib.py", line 31, in disable_importlib_metadata_finder
:info:build     to_remove = [
:info:build   File "/opt/local/var/macports/build/_private_tmp/py39-setuptools/work/setuptools-69.0.3/setuptools/_importlib.py", line 34, in <listcomp>
:info:build     if isinstance(ob, importlib_metadata.MetadataPathFinder)
:info:build AttributeError: module 'importlib_metadata' has no attribute 'MetadataPathFinder'
:info:build ERROR Backend subprocess exited when trying to invoke build_wheel

[jmroot (Joshua Root)]

Seems to have built OK on the buildbot. I take it this is a local build. Do you have py39-importlib-metadata installed?

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmroot:

Seems to have built OK on the buildbot.

Yes, py39-libxml2 succeeded on the buildbot when it was last updated three weeks ago. I thought something might have changed since then.

I take it this is a local build.

Yes, on macOS 12 x86_64.

Do you have py39-importlib-metadata installed?

Yes, but I am using trace mode and I should have added that this appears:

Warning: The following existing files were hidden from the build system by trace mode:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/importlib_metadata/__init__.py
  /private/var/select/sh

When not using trace mode, the build (of either py39-libxml2 or py39-setuptools) succeeds.

[jmroot (Joshua Root)]

OK, so if it works without trace mode whether or not py39-importlib-metadata is installed, which seems to be the case, this must be a bug in trace mode. Setuptools does use importlib_metadata but vendors its own copy, and inspects any other version that is installed to try to prevent conflicts. There have been cases previously where a file being hidden by trace mode gave different results than not having the file installed, for example #65109. That must mean that some mechanism being used to detect the file's existence is not being correctly intercepted by trace mode.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmroot:

if it works without trace mode whether or not py39-importlib-metadata is installed, which seems to be the case,

Yes I agree:

% sudo port deactivate py39-importlib-metadata
Note: It is not recommended to uninstall/deactivate a port that has dependents as it breaks the dependents.
The following ports will break:
 py39-build @1.0.3_0
 py39-sphinx @7.2.6_0
Continue? [y/N]: y
Warning: Deactivate forced.  Proceeding despite dependencies.
--->  Deactivating py39-importlib-metadata @7.0.2_0
--->  Cleaning py39-importlib-metadata
% sudo port build py39-libxml2
--->  Computing dependencies for py39-libxml2
--->  Fetching distfiles for py39-libxml2
--->  Verifying checksums for py39-libxml2
--->  Extracting py39-libxml2
--->  Applying patches to py39-libxml2
--->  Configuring py39-libxml2
--->  Building py39-libxml2
% sudo port clean py39-libxml2
--->  Cleaning py39-libxml2
%
% sudo port activate py39-importlib-metadata
--->  Computing dependencies for py39-importlib-metadata
--->  Activating py39-importlib-metadata @7.0.2_0
--->  Cleaning py39-importlib-metadata
% sudo port build py39-libxml2
--->  Computing dependencies for py39-libxml2
--->  Fetching distfiles for py39-libxml2
--->  Verifying checksums for py39-libxml2
--->  Extracting py39-libxml2
--->  Applying patches to py39-libxml2
--->  Configuring py39-libxml2
--->  Building py39-libxml2
% sudo port clean py39-libxml2
--->  Cleaning py39-libxml2
%
% sudo port -t build py39-libxml2
--->  Computing dependencies for py39-libxml2
--->  Fetching distfiles for py39-libxml2
--->  Verifying checksums for py39-libxml2
--->  Extracting py39-libxml2
Warning: The following existing file was hidden from the build system by trace mode:
  /private/var/select/sh
--->  Applying patches to py39-libxml2
Warning: The following existing file was hidden from the build system by trace mode:
  /private/var/select/sh
--->  Configuring py39-libxml2
--->  Building py39-libxml2
Warning: The following existing files were hidden from the build system by trace mode:
  /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/importlib_metadata/__init__.py
  /private/var/select/sh
Error: Failed to build py39-libxml2: command execution failed
Error: See /opt/local/var/macports/logs/_Volumes_Shared_macports-ports_python_py-libxml2/py39-libxml2/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets if you believe there is a bug.
Error: Processing of port py-libxml2 failed