Python ports do not adhere to PEP 394 and provide "python2" and "python3"

[mikko@…]

Macports provides selection of default Python interpreter as described here:

​http://stackoverflow.com/a/6159178/315168

(sudo port select python...)

It would be nice to have select for "python3" command, because Python 2.x and Python 3.x scripts are incompatible and there is no way to invoke default "python3". E.g. Ubuntu/Debian provides python3 command which points to latest Python 3.x interpreter.

Spec:

​http://www.python.org/dev/peps/pep-0394/

Example and more xplatform discussion here:

​http://stackoverflow.com/a/10164449/315168

[raimue (Rainer Müller)]

bin/python2 could be added to the normal pythongroup. However, bin/python3 would require a new select group.

[eric.lebigot@…]

Related: the python32 port contains a python3 command, which could clash with a python3 command created through port select. It looks to me like the python32 port should not contain any python3 command and that handling it should be done through port select.

+1 vote for adding a python3 group. :)

[cooljeanius (Eric Gallager)]

Cc Me!

[felipou (Felipe Machado)]

I'm interested in this too.

I'm not very experienced with creating ports (created just one so far), but it seems like there could be some complications to implementing this.

Basically it would be a matter of creating a python3 group (basically just duplicate the python_select port: ​https://trac.macports.org/browser/trunk/dports/sysutils/python_select?order=name) and setting this group at the python3 ports (like ​https://trac.macports.org/browser/trunk/dports/lang/python33/Portfile).

The thing is, I suspected it isn't possible to set multiple select groups for a single port, and python3 ports already set the python select group.

If this isn't so (that is, macports already supports multiple select groups), this seems like a fairly easy thing to do.

Otherwise, we would need to implement this in macports first. Then I won't be able to help, I have never programmed in Tcl before.

Can someone more experienced with macports enlighten us about this issue?

[felipou (Felipe Machado)]

Cc Me!

[lucractius@…]

This needs to be looked at, either the feature is easy or it is hard, can we get an answer which guides progress? Lack of python2 and python3 shortcuts leaves macports out of line with other package management tools and makes it harder to use.

[pixilla (Bradley Giesbrecht)]

The python32 port supports "port select":

$ python --version
Python 2.7.6
$ port select python
Available versions for python:
	none
	python25-apple
	python26
	python26-apple
	python27 (active)
	python27-apple
	python32
$ sudo port select python python32
Selecting 'python32' for 'python' succeeded. 'python32' is now active.
$ python --version
Python 3.2.5

[pixilla (Bradley Giesbrecht)]

Cc Me!

[seanfarley (Sean Farley)]

Replying to lucractius@…:

This needs to be looked at, either the feature is easy or it is hard, can we get an answer which guides progress? Lack of python2 and python3 shortcuts leaves macports out of line with other package management tools and makes it harder to use.

I agree. This is a historic failure of python 2.6+ not including a python2 link but is easy enough to fix in MacPorts. How do other package managers do this? How should MacPorts? I guess we should leave the current port select for python untouched and just add a python2 and python3 select group?

[seanfarley (Sean Farley)]

Replying to pixilla@…:

The python32 port supports "port select":

The issue is that there is no python2 nor python3 in ${prefix}/bin.

[pixilla (Bradley Giesbrecht)]

Replying to sean@…:

Replying to pixilla@…:

The python32 port supports "port select":

The issue is that there is no python2 nor python3 in ${prefix}/bin.

Does ​patch-python_select-add-versioned-bin.diff cover the files we would want to add {2,3} to?

[pixilla (Bradley Giesbrecht)]

If the desire is to have bin/python2 and bin/python3 side by side than new python2_select and python3_select ports could provide that.

[felipou (Felipe Machado)]

Replying to pixilla@…:

If the desire is to have bin/python2 and bin/python3 side by side than new python2_select and python3_select ports could provide that.

Can you help me do that? Just point me in the right directions.

Should I just copy the python_select port and modify what I see fit? And then request that the related python packages add the related group?

Is it ok for a port to have two "select groups" related to it? (for example, python32 would be related both to python_select and python3_select)

[aivazis@…]

Replying to pixilla@…:

Replying to sean@…:

Replying to pixilla@…:

The python32 port supports "port select":

The issue is that there is no python2 nor python3 in ${prefix}/bin.

Does ​patch-python_select-add-versioned-bin.diff cover the files we would want to add {2,3} to?

It looks to me like this patch solves the problem in its entirety. Did anyone else test? What is the progress in pushing it to distribution?

[raimue (Rainer Müller)]

Replying to aivazis@…:

Does ​patch-python_select-add-versioned-bin.diff cover the files we would want to add {2,3} to?

It looks to me like this patch solves the problem in its entirety. Did anyone else test? What is the progress in pushing it to distribution?

That won't solve the real issue. This will provide either python2 or python3, not both at the same time. As said above, we would need python2_select and python3_select groups for this.

[calvin.giles@…]

Can a select group point to the command of another select group? As in:

port select python2 python26 port select python3 python33 port select python python3

Here, python ports are divided into python2 and python3 selects, and the python select points to the results of those.

This feels cleaner to me anyway - choosing which version of python2 is separate from the 2 or 3 choice. On the other hand, it forces the python command to match the version of python2 or python3, which os a little less generic.

Can this be implemented more easiliy? (I have not done any ports dev yet)

[larryv (Lawrence Velázquez)]

Consolidating #38661, #42747, and #45130 into this ticket.

[mf2k (Frank Schima)]

Cc Me!

[myint (Steven Myint)]

Cc Me!

[anatol (Anatol Pomozov)]

Is there any progress here?

All Linux distros provide python2 and python3 links. This makes "PEP 394" really easy to follow. The only problem arises when a project should be supported on MacOSX that does not have these python{2,3} links.

It is complicated by the fact that some distros already use python3 as default (Arch), fedora 22 plans to do the same. This means there will me more and more pressure from users to use python{2,3} in script. Thus either python3 users will be hurt or (most likely) MacOSX users will be needed to update their scripts manually.

As adding python{2,3} link is easy to add and no downsize I wonder why it is still not done.

[anatol (Anatol Pomozov)]

Cc Me!

[anddam (Andrea D'Amore)]

Replying to anatol.pomozov@…:

As adding python{2,3} link is very easy and no downsize I wonder why it is still not done.

The real reason is because nobody worked at this, yet.

Patches, as always, are welcome.

[raimue (Rainer Müller)]

At the moment we only allow to control the symlink of ${prefix}/bin/python with a single select target, with which you can choose between any python2x or python3x version. However, it only allows one of the versions to be active at the same time, so it would only add python2 xor python3 symlinks.

The only solution to go forward with our current select system would be to split it into three select groups for python, python2, python3. Each of these groups would control the corresponding symlink in bin and man pages. Comment #17 above also introduced the idea to make python a choice between python2 and python3 which would also be possible, but breaks compatibility with the previous behavior.

[anatol (Anatol Pomozov)]

We follow PEP 394 in our scripts and it breaks that minority of the developers who uses macosx/macports for development. Is there any recommendation to relief the pain working with PEP-394 compliant code?

[seanfarley (Sean Farley)]

Is this as simple as providing 'port select python2' and 'port select python3'?

[felipou (Felipe Machado)]

Replying to sean@…:

Is this as simple as providing 'port select python2' and 'port select python3'?

I believe so!

[jyrkiwahlstedt]

I actually think that as the select mechanism allows one active python at any time, it would be relatively easy to provide a python2 link in python2.x versions, and python3 link in python3.x versions. Would this be enough? ('port select python2/3' isn't the same)

[larryv (Lawrence Velázquez)]

Replying to sean@…:

Is this as simple as providing 'port select python2' and 'port select python3'?

Short answer: No.

Long answer: Coming soon.

[petrrr]

Larry are you working on this?

[petrrr]

Cc Me!

[mojca (Mojca Miklavec)]

Cc Me!

[mojca (Mojca Miklavec)]

I don't know if anyone worked on this already, but one solution (in given constraints of the package manager) would be to create two new ports, python2 and python3, each of them offering variants like

• python2 [+]python27 +python27-apple ...
• python3 [+]python34 +python35 ...

and doing nothing else but installing the necessary symlinks. (And then try to safeguard against other ports trying to depend on python2 instead of python27 ;)

The advantage of separate ports over "port select python2 python27" would be the simplicity for the users (even though I agree that for consistence it would be better to extend the port select group to allow a port to belong to more than one group).

(In any case software developers probably cannot rely on existence of python2 on Mac.)

[larryv (Lawrence Velázquez)]

Replying to mojca@…:

(In any case software developers probably cannot rely on existence of python2 on Mac.)

Right. OS X itself does not conform to PEP 394.

[larryv (Lawrence Velázquez)]

Replying to larryv@…:

Long answer: Coming soon.

Okay, I lied. Sorry.

We cannot currently adhere to PEP 394 in full (not easily, anyway). But we should be able to adhere to the most important parts.

We should not simply augment the current python select group. Doing so would preclude the 2.x links and 3.x links from coexisting, since only one port can be selected at a time. We do allow multiple Python ports to be installed concurrently, so this limitation would violate a pretty important part of the PEP.

Using python2 and python3 ports would be inconsistent with our usual method of providing users with a selection. I don't think we want to create another perl5-type situation.

I recommend creating new python2 and python3 select groups.

• Pros
  • Is the standards MacPorts method of allowing users to choose a default from a selection of versioned ports.
  • Allows the 2.x and 3.x symlinks to exist concurrently.
  • Does not impose behavior on users who have already made a selection using the python group.
• Cons
  • Not actually possible right now. Surprise! The select-1.0 portgroup does not yet allow one port to register itself with multiple select groups. Fortunately, this functionality should be easy to add.
  • Cannot install the symlinks automatically whenever a python2x or python3x port is installed. MacPorts does not provide a good way of doing this.
  • Cannot force the python link to refer to the same interpreter as python2. I don't think this is important; the PEP already explicitly recommends that python be used for cross-compatible scripts only. This would, in any case, interfere with users who have already made a selection.

I will begin implementing this immediately, if no one has any compelling objections.

[larryv (Lawrence Velázquez)]

These should do it. Thanks to Ned Deily for upstream-flavored feedback.

• [136723] — Allow ports to easily install multiple select files.
• [136741,136743,136745] — Create python2 select group for Python 2.6 and 2.7.
• [136748:136749] — Create python3 select group for Python 3.3–3.5.

Selecting the default Python 2 and 3 works just like selecting the default Python always has. The three groups are independent: Changing one selection does not affect the others.

View your options:

% port select --list python
Available versions for python:
	none (active)
	python24
	python26
	python26-apple
	python27
	python27-apple
	python31
	python32
	python33
	python34
	python35
% port select --list python2
Available versions for python2:
	none (active)
	python26
	python26-apple
	python27
	python27-apple
% port select --list python3
Available versions for python3:
	none (active)
	python33
	python34
	python35

Make selections:

% sudo port select --set python python26
Selecting 'python26' for 'python' succeeded. 'python26' is now active.
% sudo port select --set python2 python27-apple
Selecting 'python27-apple' for 'python2' succeeded. 'python27-apple' is now active.
% sudo port select --set python3 python35
Selecting 'python35' for 'python3' succeeded. 'python35' is now active.

As you’d expect:

% /opt/local/bin/python -V
Python 2.6.9
% /opt/local/bin/python2 -V
Python 2.7.6
% /opt/local/bin/python3 -V
Python 3.5.0b1

The links vary by group and port. To see which links are being created, look at the debug output:

% sudo port -d select --set python3 python34
DEBUG: Copying /Users/larryv/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences
DEBUG: action_select [python3 python34] [ports_source_only yes ports_select_set yes]...
DEBUG: The 'set' command was specified.
DEBUG: mportselect [set] [python3] [python34]
DEBUG: ln -sf /opt/local/bin/python3.4 /opt/local/bin/python3
DEBUG: ln -sf /opt/local/bin/python3.4m /opt/local/bin/python3m
DEBUG: ln -sf /opt/local/bin/python3.4-32 /opt/local/bin/python3-32
DEBUG: rm -f /opt/local/bin/pythonw3
DEBUG: rm -f /opt/local/bin/pythonw3-32
DEBUG: ln -sf /opt/local/bin/python3.4-config /opt/local/bin/python3-config
DEBUG: ln -sf /opt/local/bin/python3.4m-config /opt/local/bin/python3m-config
DEBUG: ln -sf /opt/local/bin/idle3.4 /opt/local/bin/idle3
DEBUG: ln -sf /opt/local/bin/pydoc3.4 /opt/local/bin/pydoc3
DEBUG: ln -sf /opt/local/bin/2to3-3.4 /opt/local/bin/2to3-3
DEBUG: ln -sf /opt/local/bin/pyvenv-3.4 /opt/local/bin/pyvenv
DEBUG: ln -sf /opt/local/share/man/man1/python3.4.1 /opt/local/share/man/man1/python3.1
Selecting 'python34' for 'python3' succeeded. 'python34' is now active.
DEBUG: Checking time since last reclaim run

We are in the process of removing the python{24,25,31,32} ports, so they are not available as choices for the new python2 and python3 groups.

As usual, run sudo port selfupdate to receive these changes. Please open a new ticket to report bugs or request enhancements.

[anatol (Anatol Pomozov)]

Yay

$ python2
Python 2.7.10 (default, May 25 2015, 13:05:06) 
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

Thanks a lot for doing this!

[felipou (Felipe Machado)]

First of all, thanks a LOT larryv!

I just tested, and it's working great.

Although there was a strange error log when I ran these commands:

$ sudo port -d select --set python3 python34
DEBUG: Copying /Users/felipe/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences
DEBUG: action_select [python3 python34] [ports_select_set yes]...
DEBUG: The 'set' command was specified.
DEBUG: mportselect [set] [python3] [python34]
DEBUG: ln -sf /opt/local/bin/python3.4 /opt/local/bin/python3
DEBUG: ln -sf /opt/local/bin/python3.4m /opt/local/bin/python3m
Selecting 'python34' for 'python3' failed: could not create new link "/opt/local/bin/python3-32": target "/opt/local/bin/python3.4-32" doesn't exist

$ sudo port -d select --set python3 python35
DEBUG: Copying /Users/felipe/Library/Preferences/com.apple.dt.Xcode.plist to /opt/local/var/macports/home/Library/Preferences
DEBUG: action_select [python3 python35] [ports_select_set yes]...
DEBUG: The 'set' command was specified.
DEBUG: mportselect [set] [python3] [python35]
DEBUG: ln -sf /opt/local/bin/python3.5 /opt/local/bin/python3
DEBUG: ln -sf /opt/local/bin/python3.5m /opt/local/bin/python3m
Selecting 'python35' for 'python3' failed: could not create new link "/opt/local/bin/python3-32": target "/opt/local/bin/python3.5-32" doesn't exist

As I said, it worked for me, I only wanted the /opt/local/bin/python3 link, and it is correctly pointing to the chosen version. But maybe there is something broken that I didn't notice. If you want me to check anything, I'd be glad to help.

Oh, and the python2 group worked without any error messages!

Thanks again!

[larryv (Lawrence Velázquez)]

No, I know what the problem is. The Python 3 ports install python3.x-32 when built +universal. I was hoping that port select would just fail silently if those files were absent, but that clearly isn’t the case. I may have to create separate configuration files for +universal.

[larryv (Lawrence Velázquez)]

Scratch that, we’ll just drop the python3-32 link and hope that no one wants it. Run these commands to fix everything up:

% sudo port select --set python3 none          # start from scratch
% sudo port selfupdate
% sudo port upgrade outdated and 'python3*'
% sudo port -d select --set python3 python34   # or whichever you want

r136752

[felipou (Felipe Machado)]

Just tested it and I can confirm it's fixed! Thanks again!