python35 @3.5.2 missing ensurepip?

[hmijail (Horacio Mijail Antón Quiles)]

$ python3.5 -m venv myvenv  
Error: Command '['/Users/mija/myvenv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

This worked some months ago, and keeps working currently in Homebrew's python3, which is also python 3.5.2.

MacOS Sierra 10.12 (16A323), XCode Version 8.0 (8A218a)

[larryv (Lawrence Velázquez)]

Replying to hmijail@…:

$ python3.5 -m venv myvenv  
Error: Command '['/Users/mija/myvenv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

The command path suggests you’re using the interpreter from a venv. Is that what you intended? Your command works for me verbatim, using MacPorts’ Python explicitly.

% port installed python35
The following ports are currently installed:
  python35 @3.5.2_0+universal (active)
% cd /private/tmp
% /opt/local/bin/python3.5 -m venv myvenv
% ls myvenv
bin/		include/	lib/		pyvenv.cfg
%

[hmijail (Horacio Mijail Antón Quiles)]

I think that this is not what is happening.

1:~$ mkdir test
2:~$ cd test
3:~/test$ python3.5 -m venv venv
Error: Command '['/Users/mija/test/venv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

My understanding is that venv creates the virtual environment and immediately goes into it and installs ensurepip. And I guess that in this last step is where the ensurepip installation is failing.

In fact, adding --without-pip it works.

$ python3.5 -m venv --without-pip venv2
$

[hmijail (Horacio Mijail Antón Quiles)]

Just to make sure, I just uninstalled the python35 port and installed it again (I already tried that a couple of days ago, but who knows).

The problem is still here.

[larryv (Lawrence Velázquez)]

I suspect there is an issue with your configuration. I am having no trouble with this at all.

Are you sure python3.5 is resolving to MacPorts’ Python? What does this return?

$ command -v python3.5

What happens when you create a venv the recommended way?

$ cd /private/tmp && /opt/local/bin/pyvenv-3.5 testenv

[hmijail (Horacio Mijail Antón Quiles)]

1:~$ command -v python3.5
/opt/local/bin/python3.5
2:~$ cd /private/tmp && /opt/local/bin/pyvenv-3.5 testenv
Error: Command '['/private/tmp/testenv/bin/python3.5', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1
3:/private/tmp$

I actually don't discard some local configuration problem, since I'm seeing very strange breakage with pyinvoke installed with pip3. However, I don't see where the breakage can come from, and I still didn't have the time to dig into that.

I don't usually work with Python, by the way.

[larryv (Lawrence Velázquez)]

Replying to hmijail@…:

I don't usually work with Python, by the way.

Neither do I :)

Let’s try something. Do you see any output if you run that command manually?

$ cd /private/tmp/testenv && /private/tmp/testenv/bin/python3.5 -Im ensurepip --upgrade --default-pip; echo $?

[hmijail (Horacio Mijail Antón Quiles)]

(I just saw your request when I was about to post this; will do it in a second)

Just to make sure that I wasn't imagining things, I just uninstalled the python35 port, installed Homebrew's python3 (Python 3.5.2, formula version3.5.2_2), and it works OK.

10:/private/tmp$ /usr/local/bin/pyvenv-3.5 testenv2
11:/private/tmp$

In case it matters: I upgraded to macOS Sierra a few days ago, and I followed the MacPorts migration instructions (delete everything, install new MacPorts). I didn't do anything special with brew, since I didn't see any instructions; and in fact it all mostly seems to work, apart from some permissions problems.

[hmijail (Horacio Mijail Antón Quiles)]

(Regarding the migration, in fact I remember now that after deleting everything following the migration instructions, there were still a lot of symlinks under /opt. I didn't delete them manually, assuming they would get overwritten as needed and shouldn't be a problem...)

3:/private/tmp$ command -v python3.5
/opt/local/bin/python3.5
4:/private/tmp$ cd /private/tmp/testenv && /private/tmp/testenv/bin/python3.5 -Im ensurepip --upgrade --default-pip; echo $?
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__init__.py", line 209, in _main
    default_pip=args.default_pip,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__init__.py", line 116, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ensurepip/__init__.py", line 41, in _run_pip
    pip.main(args)
  File "/var/folders/2d/1kly9t3s23bclpxgs6l342vc0000gs/T/tmplhya0ke0/pip-8.1.1-py2.py3-none-any.whl/pip/__init__.py", line 215, in main
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/locale.py", line 594, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting
1

OMFG, locales?

I just recently moved into iTerm2, which has its own locale settings that I didn't explore yet. And it's known that Python has problems with non-GNU locales ​https://bugs.python.org/issue18378 , so maybe that's it.

(though that doesn't explain the problems with pyinvoke...)

[hmijail (Horacio Mijail Antón Quiles)]

Well, I'm running Homebrew in the same iTerm2, so that can't be it all.

[hmijail (Horacio Mijail Antón Quiles)]

For reference: the locale settings I have in iTerm are these:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The ones I have in Terminal are:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Anyway, in Terminal the failures are the same as in iTerm.

[jmroot (Joshua Root)]

Not really sure why I'm on Cc here, but…

LC_CTYPE="UTF-8"

Pretty sure that's invalid. You can only use one of the values listed by locale -a. I vaguely recall some other bug where unvalidated environment variables being passed straight to setlocale caused problems (meld?).

Also, it looks like this may have been fixed in pip 8.1.2: ​https://github.com/pypa/pip/issues/3575

[hmijail (Horacio Mijail Antón Quiles)]

I didn't set the locales like that. Terminal.app sets them like that when the checkbox [Preferences/Profiles/Advanced/Set locale environment variables on startup] is unchecked. iTerm sets them like that when the checkbox [Preferences/Profiles/Terminal/Set locale variables automatically] is checked. Meaning, that it is some kind of "standard", not some configuration I made myself. According to the python bug report I posted, this seems rather to be a question of BSD vs GNU.

Anyway, in both configurations, Homebrew's Python works and MacPorts' doesn't.

[larryv (Lawrence Velázquez)]

Replying to jmr@…:

Not really sure why I'm on Cc here, but…

Domain expertise? :)

[larryv (Lawrence Velázquez)]

Replying to hmijail@…:

I didn't set the locales like that. Terminal.app sets them like that when the checkbox [Preferences/Profiles/Advanced/Set locale environment variables on startup] is unchecked. iTerm sets them like that when the checkbox [Preferences/Profiles/Terminal/Set locale variables automatically] is checked. Meaning, that it is some kind of "standard", not some configuration I made myself. According to the python bug report I posted, this seems rather to be a question of BSD vs GNU.

Have you fiddled with your Language & Region settings?

• comment:9:ticket:52420
• ​https://bugs.python.org/issue18378#msg215215

Anyway, in both configurations, Homebrew's Python works and MacPorts' doesn't.

Looks like they ​grab pip 8.1.2 separately.

[jmroot (Joshua Root)]

Yeah, this is all laid out in the upstream bug linked in comment:9. Apple does something different to everyone else (but still conforms to the spec), python doesn't handle it well. The bundled version of pip in python35 doesn't have a workaround for the problem.

The workaround is easy, set your locale environment variables to something that python likes.

[hmijail (Horacio Mijail Antón Quiles)]

As already noted, venvs worked well with MacPorts's Python 3.5 from a couple of months ago, just as it does now with Homebrew's current Python.

[larryv (Lawrence Velázquez)]

And you’re sure you haven’t changed any Language & Region settings since then?

[hmijail (Horacio Mijail Antón Quiles)]

As far as I can remember, I didn't change that, exactly because of the python problems: I found that disabling Terminal's location setting fixed some problem and left it at that.

On the other hand, as stated, I updated to macOS Sierra a few days ago, so I don't know if Apple changed some Terminal default.

The only reason iTerm is still using its enabled-by-default "set locale" pref is because I'm still getting into it and didn't want to change too many preferences at once.

[hmijail (Horacio Mijail Antón Quiles)]

Looks like they ​grab pip 8.1.2 separately.

But this problem is about ensurepip, not about pip, which AFAICT is in its own MacPorts package, no? (py35-pip)

[hmijail (Horacio Mijail Antón Quiles)]

Homebrew's python3's ensurepip is 8.1.1:

6:~/test$ python3 -m venv venv3
7:~/test$ . venv3/bin/activate
(venv3) 8:~/test$ pip --version
pip 8.1.1 from /Users/mija/test/venv3/lib/python3.5/site-packages (python 3.5)
(venv3) 9:~/test$

[hmijail (Horacio Mijail Antón Quiles)]

I mentioned strange breakage in pyinvoke. That ended up being a pykerberos problem in Python 3 (general, happened the same on Homebrew), being worked on. So the only strange thing remaining is in MacPorts' Python3's ensurepip. (Just confirmed in python34 too)

[hmijail (Horacio Mijail Antón Quiles)]

In case it helps anyone in the future: a way to fix MacPorts' Python3's ensurepip is

$ export LC_ALL=en_US.UTF-8
$ python -m venv mpvenv
$

...or instead of en_US.UTF-8, use C, or maybe any other option shown by locale -a.

...or just go with Homebrew, which seems to just work with the default configurations of... everything?

[larryv (Lawrence Velázquez)]

Replying to hmijail@…:

Homebrew's python3's ensurepip is 8.1.1:

That’s pip, not “ensurepip”. And I guess I don’t understand what that bit about 8.1.2 in the formula was about then.

​This bit is curious. Does this work with Homebrew’s Python 3?

$ python3.5 -c 'import ensurepip'

[hmijail (Horacio Mijail Antón Quiles)]

That’s pip, not “ensurepip”. And I guess I don’t understand what that bit about 8.1.2 in the formula was about then.

Why do you say it's not ensurepip, to begin with?

[larryv (Lawrence Velázquez)]

ensurepip is a ​module that installs pip. It’s part of CPython; it doesn’t have a version. pip --version invokes pip.

[hmijail (Horacio Mijail Antón Quiles)]

ensurepip is a ​module that installs pip. It’s part of CPython; it doesn’t have a version. pip --version invokes pip.

PEP 453 seems to disagree. -m ensurepip is an interface for cases where pip was not installed or uninstalled. "In most cases, end users won't need to use this CLI directly, as pip should have been installed automatically when installing Python or when creating a virtual environment. However, it is formally documented as a public interface to support at least these known use cases: ..."

"ensurepip itself (including the private copy of pip and its dependencies) will always be installed normally (as it is a regular part of the standard library), but an option will be provided to skip the invocation of ensurepip"

Now, PEP 453 also seems to say that ensurepip does nothing if an updated pip is installed.

​https://www.python.org/dev/peps/pep-0453/

[hmijail (Horacio Mijail Antón Quiles)]

Homebrew's python3:

$ python3 -c 'import ensurepip'
$ echo $?
0

[larryv (Lawrence Velázquez)]

I don’t know what you’re trying to prove. I misspoke about the installing, but ensurepip still doesn’t have a “version”. It contains a private copy of pip, which itself has a version, which for Python 3.5.2 is 8.1.1.

Anyway I’m at a loss. The bug is in Python, not MacPorts. Homebrew’s packaging must be avoiding the problem somehow, but I cannot figure out what the formula is doing through all the Ruby syntax. And since it’s ultimately an unresolved upstream bug, I’m not really convinced that we should be trying to fix it.

Cc-ing an upstream dev to see if he has any ideas.

[hmijail (Horacio Mijail Antón Quiles)]

I also don't know what you are trying to prove. Even MacPorts' python35's ensurepip (a) answers as pip, (b) does have a version number.

$ export LC_ALL=C
$ which python3.5
/opt/local/bin/python3.5
$ python3.5 -m venv venv10
$ . venv10/bin/activate
(venv10) $ pip --version
pip 8.1.1 from /Users/mija/test/venv10/lib/python3.5/site-packages (python 3.5)
(venv10) $ port installed py35-pip
The following ports are currently installed:
  py35-pip @8.1.2_0
$

[larryv (Lawrence Velázquez)]

I am trying to get us on the same page with the terminology. That program you are running inside the venv is the private copy of pip provided by ensurepip. It is not ensurepip itself. Look inside, ensurepip is not to be found.

#!/private/tmp/myvenv/bin/python3.5

# -*- coding: utf-8 -*-
import re
import sys

from pip import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

[hmijail (Horacio Mijail Antón Quiles)]

Oh, OK, I see now what you meant. Sorry, my fault.

[ned-deily (Ned Deily)]

The only thing I can add to the discussion at the moment is that I can reproduce the OP's ensurepip failure when LC_CTYPE=UTF-8 and using the venv from the current MacPorts python35 (3.5.2). However, I don't see a similar failure when using the python3.5 from the python.org 3.5.2 installer for OS X which has the same bundled versions of pip (8.1.1) and setuptools (20.10.1). I took a very quick look at the current MacPort python35 patches and didn't see anything obvious. I'm afraid I don't have time right now to dig in any deeper. But, obviously as noted above, the easy workaround is to ensure locale variables are set to more standard values. I just throw in an export LANG=en_US.UTF-8 in my .bashrc.

[jmroot (Joshua Root)]

If this wasn't fixed in python 3.5.x at some point, I guess it never will be.