py312-virtualenv@20.26.2 is incompatible with Python 3.12

[Themanwithoutaplan]

I'm really not exactly sure where this is coming from but an older version of setuptools gets installed which is itself not compatible with Python 3.12

Here's how to replicate it

 virtualenv-3.12 test                                                                           [16:06:16]
created virtual environment CPython3.12.3.final.0-64 in 519ms
  creator CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
    added seed packages: pip==22.3.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

cd test
pip list

This produces the following error message

ERROR: Exception:
Traceback (most recent call last):
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/commands/list.py", line 169, in run
    packages: "_ProcessedDists" = [
                                  ^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 664, in <genexpr>
    return (d for d in it if d.canonical_name not in skip)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Traceback (most recent call last):
  File "/Users/charlieclark/temp/test/bin/pip", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 70, in main
    return command.main(cmd_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 101, in main
    return self._main(args)
           ^^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    self.handle_pip_version_check(options)
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 179, in handle_pip_version_check
    session = self._build_session(
              ^^^^^^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/network/session.py", line 343, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/network/session.py", line 175, in user_agent
    setuptools_dist = get_default_environment().get_distribution("setuptools")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 188, in get_distribution
    return next(matches, None)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 183, in <genexpr>
    matches = (
              ^
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 612, in iter_all_distributions
    for dist in self._iter_distributions():
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 176, in _iter_distributions
    for dist in finder.find_eggs(location):
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 144, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 111, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "/Users/charlieclark/temp/test/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

I'm not sure whether the problem is being pinned to pip==22.3.1 or another dependency upon an older version of setuptools.

[reneeotten (Renee Otten)]

After you create a virtual environment, you have to activate it first - from the commands you show in the ticket here that is not what you did. Just cd-ing to the directory is not going to work.

When I try the following locally, it works fine for me:

virtualenv-3.12 /tmp/test
source /tmp/test/bin/activate
pip list

[Themanwithoutaplan]

Of course I did activate the venv… Here's a one-liner:

virtualenv-3.12 test && source test/bin/activate && pip list with the same result.

I've reinstalled python312 to no effect. Can you include what version of pip gets installed/seeded on your system? I have actually 24.0 but I'm still seeing 22.3.1 being installed in the venv, because I think this is the actual problem.

[reneeotten (Renee Otten)]

Replying to Themanwithoutaplan:

Of course I did activate the venv… Here's a one-liner:

well, you provide code to reproduce the presumed problem but didn't do it there - so how am I supposed to know what you did if you don't mention it...

virtualenv-3.12 test && source test/bin/activate && pip list with the same result.

I've reinstalled python312 to no effect. Can you include what version of pip gets installed/seeded on your system? I have actually 24.0 but I'm still seeing 22.3.1 being installed in the venv, because I think this is the actual problem.

that results in

❯ pip list
Package Version
------- -------
pip     24.0

so again, it works all as expected.

Please check what version of py312-virtualenv you have installed, that should be 20.26.2_0.

[jmroot (Joshua Root)]

Do you have a system-wide or per-user virtualenv.ini? Is VIRTUALENV_CONFIG_FILE set in the environment? The default configuration won't install a seed setuptools at all on python 3.12.

[Themanwithoutaplan]

This is what I have installed:

  py312-pip @24.0_0 (active)
  py312-platformdirs @4.2.1_0
  py312-platformdirs @4.2.2_0 (active)
  py312-pycryptodome @3.19.0_0 (active)
  py312-setuptools @69.5.1_0 (active)
  py312-toml @0.10.2_0 (active)
  py312-virtualenv @20.26.2_0 (active)

And I don't have any relevant config files and have been using MacPorts to drive multiple Pythons (with tox) for years.

I have no problems if I use python3.12 -m venv test

But look at what happens if I pass the -v flag:

virtualenv-3.12 -v test                                                                        [12:59:37]
find interpreter for spec PythonSpec(path=/opt/local/Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12)
proposed PythonInfo(spec=CPython3.12.3.final.0-64, exe=/opt/local/Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12, platform=darwin, version='3.12.3 (main, Apr 12 2024, 17:19:53) [Clang 14.0.3 (clang-1403.0.22.14.1)]', encoding_fs_io=utf-8-utf-8)
create virtual environment via CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
triggered periodic upgrade of pip==24.0 (for python 3.12) via background process having PID 29207
add activators for Bash, CShell, Fish, Nushell, PowerShell, Python
created virtual environment CPython3.12.3.final.0-64 in 946ms
  creator CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
    added seed packages: pip==22.3.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

charlieclark:temp/ $ virtualenv-3.12  test                                                                          [13:00:01]
created virtual environment CPython3.12.3.final.0-64 in 473ms
  creator CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
    added seed packages: pip==22.3.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

With it, pip@24.0 gets installed, without it, pip@22.3.1. That's what I called weird™.

I've just tried uninstalling virtualenv from MacPorts and installing it as my user as that seeding directory makes me suspicious:

pip-3.12 install virtualenv                                                                    [13:03:40]
Defaulting to user installation because normal site-packages is not writeable
Collecting virtualenv
  Using cached virtualenv-20.26.2-py3-none-any.whl.metadata (4.4 kB)
Requirement already satisfied: distlib<1,>=0.3.7 in /opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from virtualenv) (0.3.8)
Requirement already satisfied: filelock<4,>=3.12.2 in /opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from virtualenv) (3.13.1)
Requirement already satisfied: platformdirs<5,>=3.9.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages (from virtualenv) (4.2.2)
Using cached virtualenv-20.26.2-py3-none-any.whl (3.9 MB)
Installing collected packages: virtualenv
  WARNING: The script virtualenv is installed in '/Users/charlieclark/Library/Python/3.12/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed virtualenv-20.26.2
charlieclark:temp/ $ ~/Library/Python/3.12/bin/virtualenv test                                                      [13:04:47]
created virtual environment CPython3.12.3.final.0-64 in 496ms
  creator CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
    added seed packages: pip==22.3.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

charlieclark:temp/ $ ~/Library/Python/3.12/bin/virtualenv -v test                                                   [13:05:34]
find interpreter for spec PythonSpec(path=/opt/local/Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12)
proposed PythonInfo(spec=CPython3.12.3.final.0-64, exe=/opt/local/Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12, platform=darwin, version='3.12.3 (main, Apr 12 2024, 17:19:53) [Clang 14.0.3 (clang-1403.0.22.14.1)]', encoding_fs_io=utf-8-utf-8)
create virtual environment via CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
add activators for Bash, CShell, Fish, Nushell, PowerShell, Python
created virtual environment CPython3.12.3.final.0-64 in 567ms
  creator CPython3Posix(dest=/Users/charlieclark/temp/test, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/charlieclark/Library/Application Support/virtualenv)
    added seed packages: pip==22.3.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

So, the problem can't be with MacPorts (I never thought it could be). The timestamp on this directory is 2020, which is when I got this computer. I removed it and lo and behold, pip@24.0 gets installed.

This folder gets created by virtualenv itself. I wonder what's causing it to go stale?

Any sorry for the noise and thanks for your help!