Install of youtube-dl requires multiple Python modules of various versions

[spmundi]

I am running MacPorts 2.6.2 I have python36 installed and selected.

I want to install youtube-dl. The install is asking me to install python27, python38 and various python37 modules. Here is a list: ffmpeg gdk-pixbuf2 gobject-introspection graphviz ld64 ld64-xcode libLASi librsvg nasm pango py37-beaker py37-mako py37-markdown py37-markupsafe py37-setuptools python27 python37 python38 vala I do not understand how the dependencies can be requiring versions of Python that, even if installed, would not be the active versions. Evidently the sub-Portfiles of youtube-dl, aren specifying dependencies on these python versions. But why? Should not that depency be merely on python and not a specific version of python?

It does no good for me to try and install using variants. EG port install youtube-dl -python38. There is no difference in the output. I do not wish to have me ports cluttered up with all manner of pythonx.x installs which I am not using.

[kencu (Ken)]

looks like clang-3.4 is calling in python27, at least on this system I'm on right now. clang-3.4 can't, I believe, use anything in the python3.X series, so you're stuck there.

youtube-dl has chosen to use python38 as it's default variant. However, you can just use python37 if you install youtube-dl with the +python37 variant.

your message says

It does no good for me to try and install using variants. EG port install youtube-dl -python38. There is no difference in the output. I do not wish to have me ports cluttered up with all manner of pythonx.x installs which I am not using.

I have no idea what you are trying to actually say there.

But if you "I do not wish to have me ports cluttered up with all manner of pythonx.x installs which I am not using." then you might want to install youtube-dl with the python37 variant.

That is pretty much the best we can do. If that's not sufficient for you, as they say "pull requests are welcome" to improve the situation, and if you should come up with something useful, they will be reviewed and critiqued in detail, as always.

[spmundi]

Surely modules should not depend on minor versions of python should they?

You have a situation here where I would happily just accept the python38 module, but because gobject-introspection (referenced somewhere in the dependency tree) has explicit dependencies on py37-mako and py37-markdown, I am forced to drag in the whole of python37 as well (I still don't know where py37-beaker is referenced)

I realise these interdependencies become a whole rat's nest and that python has its own method of resolving dependencies (another boondoggle with conda, pip, pyenv, pipenv ..... another mess) which also complicates the issue. But surely modules should not depend on minor releases of python (unless it has to be >= a certain minor release).

Are you, or better said is the creator of the Portfile, telling me that youtube-dl explicitly invokes, eg

/opt/local/bin/markdown_py-3.7

or that it knows how to reference python modules deep within this path:

/opt/local/Library/Frameworks/Python/Versions/3.7/lib/python3.7/site-packages/markdown

even when the current version of python selected by "port select" is 3.6 (or 3.8 or 2.7)?

[kencu (Ken)]

We do have two competing influences here to try to optimize. On the one hand, we should try to minimize install volume. So if any old python would suffice, we should not be overly picky about which one is used. On the other hand, for ReproducibleBuilds, we like to fully specify every detail of the build, including the exact python used. It's a balance, but when in doubt, we error to the side of ReproducibleBuilds, as that is usually safer and easier to debug.

I have also thought that it would be useful if MacPorts might be able specify a floor version of a certain port, like python >= 3.5 for example, and then assess somehow to see what is installed on a given system, and use it if it's new enough, otherwise install something if it is not new enough. That might solve a lot of what you're noticing.

So far, we don't have that capability written into MacPorts, however, and as it is a step in the opposite direction to ReproducibleBuilds we may not get much purchase on the idea as a whole.

If you install youtube-dl with the python37 variant you'll solve most of your frustration -- please just do that, I suggest.

[ryandesign (Ryan Carsten Schmidt)]

In MacPorts, each X.Y version of Python is its own separate world. We do not have a feature where a port could specify that it requires a particular minimum version of Python. If a port requires Python but does not care at all how old it is, it can just use the macOS-provided copy of Python, but if the port has any version restrictions on Python whatsoever (for example if it requires Python 3, or if it requires Python 2.6 or later), it must specify exactly which X.Y version it wants. Also, any port that requires any Python module not provided in macOS must specify which X.Y version of Python shall provide those modules.

Often in these cases the port will offer variants so the user can choose, and the default variant should be reasonable—for example the latest stable version. But when a new version of Python is released, everyone must notice that this has occurred and update every port's default variant.

youtube-dl offers these variants, and has already updated its default variant to python38. Probably some of its dependencies have not, and should do so. If you notice ports that are defaulting to a python3 older than python38, you can file tickets or pull requests to get them updated to python38.

There's nothing to fix about the youtube-dl port so I'll close this ticket. If you have further questions about why MacPorts does what it does or have suggestions about how we should change things, please write to our mailing lists.