Executables installed by pip are not linked into macports path (and are not “select”able in any way).

[gallafent]

There's some prior discussion of this problem at ​https://github.com/pypa/pip/issues/113

In short: when I install a python package using pip, its executables are installed into the appropriately versioned python tree, but those executables are not symlinked to somewhere which is on my PATH.

In my example, I have installed macports' pip, and have port selected python35 and pip35. I then used pip to install the aws cli tools, sudo pip install aws. The aws binary is installed by pip at /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/aws. So far so good. The problem is that that path is not on my PATH (of course … port select can't alter my path, only the individual symlinks in /opt/local/bin to the objects which are owned by that particular port for which port select was run).

I would naively expect at first glance that issuing a sudo port select pip pip35 would cause any binaries subsequently installed by that version of pip to be linked into my macports path /opt/local/bin. Of course that doean't happen with a standard pip. There follows the question of how to achieve this, since standard pip doesn't know it's supposed to do something about any binaries installed /after/ the port select operation, other than to install them in the appropriate place in the python heirarchy for the selected version of pip. This puts the question firmly in MacPorts' world, as per the latest comments on the github issue.

Beporter's comment (third from last in that github issue at the time of writing) echos my feeling here … I would expect that something installed with the currently selected version of pip would end up on my path, though I completely understand why it doesn't at present.

There's clearly some precedent for linking binaries to do with a specific python version directly in this way, since for example idle is linked thus on my installation, so the desired end result (I end up with a link /opt/local/bin/aws -> /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/aws) would be consistent with this:

idle -> /opt/local/bin/idle3.5
idle2.7 -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/idle2.7
idle3 -> /opt/local/bin/idle3.5
idle3.4 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.4/bin/idle3.4
idle3.5 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/idle3.5

One way to achieve this (the “best” that I can think of so far) would be to modify MacPorts' pip so that when a given version of pip installs a binary it also links that into /opt/local/bin. This shouldn't be too hard to achieve but it's pretty invasive w.r.t. pip's behaviour. It would, though, solve the oddness seen by me and others leading to pip-installed executables not appearing on my path. The port select pip pipXX command would also need on each invocation either to invoke pip or examine pip's registry in order to unlink the executables which were installed by the version of pip being deselected, and link those owned by the version of pip being selected. Not rocket science, but a bit complicated.

The alternative would seem to be to provide stub ports of some sort which use pip as the installation method under the hood but which handle the symlinking by providing port select machinery for each such port. That feels like a maintenance drag though (keeping up with pip's repositories …)

(Footnote: I can see that there's an existing port py-awscli which should provide what I need for now specifically in my example aws case, though it's pretty outdated, but that doesn't negate the general point about pip and paths in this bug …)

[gallafent]

Apologies, submitted too soon, forgot to CC maintainer stromnov@…, and can't see a way to do that after the event! Likewise I can't change the port field from the nonexistent pip to something which does exist, i.e. py35-pip!

[stromnov (Andrey Stromnov)]

It's a rather bad practice to mix up several different package managers in one place (port and pip). If you, for example, manually install tornado using macports pip-3.5, then you can entirely break your macports installation due to dependency chain update (setuptools, certify and so on).

So, it's preferably to install custom packages into separate dirs, or install custom packages using macports build system.

[gallafent]

Replying to stromnov@…:

So, it's preferably to install custom packages into separate dirs, or install custom packages using macports build system.

I guess the “correct” response to the absence of a given package in macports is to request that it be added then (e.g. I might want there to exist a package py35-awscli (purely theoretically ;), in which case I should request that new port. It seems a pity not to be able to use the pip infrastructure in a more synergistic way with macports, but I guess we are where we are!

This does ask the rather more serious question though: if it's bad practice to install packages in macports python using pip, why does macports provide a port of pip? That's a discussion for another day though!

[stromnov (Andrey Stromnov)]

The same situation exists in any other *nix-like OS: ​http://askubuntu.com/questions/431780/apt-get-install-vs-pip-install

You can use pip to install custom packages in any other location than system-wide (using --user or --target options).

[hmijail (Horacio Mijail Antón Quiles)]

The "you can entirely break your macports installation" comment was scary, so I'd like to make sure that I got it right:

One should NOT use pip to install packages on the macports directory tree (which is what a macports-installed pip would do by default!). The recommended procedure would be:

• Install with macports your python distribution
• if that python doesn't have its own standard virtualenv and pip (like Python 3.4 and later include venv and ensurepip), then install with macports the appropriate pip and virtualenv for that python
• Prepare a "new place" to install pip packages (like a virtualenv)
• Once into the "new place", use pip to install whatever

A possible, but risky exception, would be: macports not carrying a pip-available package that has to be installed globally. That package could then be installed through pip into the macports directory tree, without fear of causing a conflict - until the day that macports *does* carry that package, and then the risk of conflict appears. Right?

Is this documented somewhere? I'd say it should!

[hmijail (Horacio Mijail Antón Quiles)]

Cc Me!

[SickTeddyBear]

Cc Me!

[wladston (Wladston Ferreira Filho)]

Cc Me!

[wladston (Wladston Ferreira Filho)]

Cc Me!

[cogentParadigm (Ali Gangji)]

I am dynamically referencing the correct bin directory based on the currently active version of python using this line in my .bash_profile:

export PATH="$PATH:/opt/local/Library/Frameworks/Python.framework/Versions/$(stat -f %Y /opt/local/bin/python | sed 's/[^0-9\.]*//g')/bin"

This won't automatically update your current shell when changing the active version of python. You'll have to source .bash_profile or start a new shell.

[fmw42 (Fred Weinhaus)]

I apologize. I am not a software engineer. I do not understand this ticket. Is it really saying not to use MacPorts to installed pip in order to for example install opencv and other python used support tools such as numpy, etc? I have never used pip before and have relied upon MacPorts to install Python/OpenCV/Numpy/etc.

I need to use Python and OpenCV. But the MacPorts direct install for OpenCV for my M1 Mac Ventura does not work. It has errors when trying to install it and there has been no progress to my knowledge in getting this fixed since I posted the following tickets. See #69381 and #69414.

If one should truly not use MacPorts install of pip, what is the recommended process to install pip compatible with MacPorts? I don't want two separate repositories of my needed tools - MacPorts and Pip.

Any help would be greatly appreciated.

[ryandesign (Ryan Carsten Schmidt)]

Don't use MacPorts pip with sudo to install python modules into the MacPorts prefix; use MacPorts py-* ports to do that. Do use MacPorts pip without sudo to install modules into project-specific directories for use with projects outside of MacPorts.

[fmw42 (Fred Weinhaus)]

I was not planning to use sudo. I simply want to install Python and pip with MacPorts and then use pip to install OpenCV and Numpy and other things like Scipy, PIL, and some other Python tools that are not currently supported by MacPorts yet . But I am not sure how to handle the situation that those tools have dependencies that would be automatically installed by pip. However, I also install Imagemagick from source after installed my needed delegates with MacPorts. Some of the same dependencies are used by Imagemagick as with Python via Pip (and need to be in a directory in my PATH environment variable to be found by Imagemagick.

How do I avoid having duplicates? Is there a way to have only one repository for them?

[hmijail (Horacio Mijail Antón Quiles)]

Replying to fmw42:

I am a software engineer but not a full-time Python programmer, and I still feel your pain. I would recommend you to follow this guide: ​https://www.bitecode.dev/p/relieving-your-python-packaging-pain

In short, don't try to use a package manager's python. That only gives you more chances for pain. Use the official installers, and keep your stuff totally separate from MacPorts'.

Note that this also applies to other package managers like homebrew. Their python is rather meant for their own packages, not really for you as a python user; and you should be using virtual environments anyway.

Everything is explained in that guide, both at the basic survival level and at the deep reasoned level.

[ryandesign (Ryan Carsten Schmidt)]

Fred, if you don't use sudo when you use pip then there is no danger of you inadvertently installing modules into the MacPorts prefix so that's fine. ImageMagick 6 and 7 are available in MacPorts so there's no need for you to build it from source.

Horacio, I can't speak for Homebrew or the contents of the article you linked to, but MacPorts ports, including pythons, are (of course) intended for use by users. There's no need to install a separate copy of python outside of MacPorts that you then have to separately remember to update.

[fmw42 (Fred Weinhaus)]

Ryan,

Thanks for you answer.

News to me that MacPorts now supports Imagemagick 7. When did that happen?

But what if I need an Imagemagick delegate that is not currently installed with Imagemagick? The downside is that I have to wait for MacPorts to update Imagemagick whose release often is upgraded weekly (and I do testing and answer Imagemagick questions, so need to be current). That was why I installed via source but use MacPorts to get my delegates.

If I use MacPorts pip to install Python related tools, but use MacPorts for my Imagemagick delegates, then I would have duplicates of a number of things. That was what I wanted to try to avoid. That would happen whether I would use MacPorts Pip or Pip installed for my Mac Python if I want to continue installing Imagemagick from source with MacPorts delegates. Is my logic correct? Or is there some way to have only one set.

P.S. What is taking so long to fix the MacPorts OpenCV?

[hmijail (Horacio Mijail Antón Quiles)]

Replying to ryandesign:

Horacio, I can't speak for Homebrew or the contents of the article you linked to, but MacPorts ports, including pythons, are (of course) intended for use by users. There's no need to install a separate copy of python outside of MacPorts that you then have to separately remember to update.

Ryan, my comment was informed in part by the fact that I also commented in this very same issue 8 years ago with my difficulties (see comment:6). My comment never got addressed. Eventually I dealt with it, because fortunately I can debug stuff myself.

I appreciate MacPorts wants to be used by Python users - would be nice. I wonder what it does better than Homebrew for that to make sense. 8 years later, that guide for beginners still resonates too much. I am not trying again, and personally I'm not recommending others they do.