p5* ports: thoughts, observations, questions

[RJVB (René Bertin)]

Sorry for the title, but someone has to say it. I've hesitated between "defect", "enhancement" and "request" for the ticket type but went with "defect" in the end. Feel free to change that.

Regularly I find that perl5 ports I have installed and depended on on 1 system have disappeared when I try to install them on another system. There used to be a time (IIRC) where one could just depend on port:perl5 and whatever port:p5-foo were required. That's probably still the case for the main port but it just doesn't work anymore for the others. What's the rationale for not including all versions from the main port in the p5-foo ports? For instance, in the snapshot of the main repo I have I see that port:perl5 has Sub-ports: perl5.16, perl5.18, perl5.20, perl5.22, perl5.24, perl5.26, perl5.28, perl5.30, perl5.32, perl5.34, perl5.36 but the p5-foo ports have perl5.branches 5.28 5.30 5.32 5.34 . That can't be right (and frankly, that variable should be set in a PG and only overridden in individual Portfiles when unavoidable).

Also, I see the perl5 PG now misdefines perl5.major as major.minor, which is probably part of the reason of the mess.

I am aware that some of the p5.xy-foo ports install binary shared libraries, but as far as I can tell the ones I have installed on this machine do not link to libraries provided by the corresponding port:perl5.xy.

I do not see any evidence that perl is included in the port select mechanism.

In other words, wouldn't it be much cleaner to provide a port:perl5 that is not just a dependent but installs an actual default version of perl5, which gets updated regularly but not more often than strictly necessary, and which can be used preferentially for things like build dependencies? Updating that main-default port would only have to trigger a rebuild of the corresponding p5-foo ports when there's been a ABI break (very rare, I assume since Linux distros do not seem to maintain separate lineages of perl packages).

This would address the issues I'm referring to above, and also put an end to (a priori) the ever-increasing amount of obsolete p5-foo ports that never get cleaned up if the user doesn't do it because "who knows what they're needed for".

Having such a main/default perl5 version is not incompatible with having a number of specific versions with their corresponding p5-foo ports, for users who have specific needs.

---

Of course it could also be an idea to allow advanced users to set a preferred perl5 version in macports.conf (which would get included in perl5.branches) - and let the perl5 PG print out a warning when it's really time to bump that version.

[ryandesign (Ryan Carsten Schmidt)]

Replying to RJVB:

Sorry for the title, but someone has to say it.

Yeah, it's not particularly conducive to a productive discussion. It's provocative and puts everyone on the defensive to begin with. Calling it a mess is disrespectful to the maintainers who have invested so much of their unpaid time into making it work as well as it does (and in my opinion it works very well).

Your observations would work better as a mailing list discussion.

Regularly I find that perl5 ports I have installed and depended on on 1 system have disappeared when I try to install them on another system.

If you mean older perl versions of modules, yes, it has been a regular habit of the perl contributors to remove modules for several-years-old perl versions in an effort to clean things up.

Other strategies are possible. In the php module ports I choose to keep obsolete versions available. But I don't maintain the perl ports so I'm not going to complain when someone else does. As I recall, it was claimed that perl was mature software and new versions are usually very compatible with the previous version such that there isn't as much need for keeping the old versions around as there is with languages that are still undergoing more substantial changes like python or php.

There used to be a time (IIRC) where one could just depend on port:perl5 and whatever port:p5-foo were required. That's probably still the case for the main port but it just doesn't work anymore for the others.

You can depend on port:perl5 if you require perl, don't care what version it is, and don't use any modules. You can depend on bin:perl:perl5 if you are OK with the perhaps considerably older perl provided in macOS. If you need modules, you must depend on the specific perl version of the modules you want. It has been this way since perl5 and the p5- ports were split into submodules in 2011 (#30638).

What's the rationale for not including all versions from the main port in the p5-foo ports? For instance, in the snapshot of the main repo I have I see that port:perl5 has Sub-ports: perl5.16, perl5.18, perl5.20, perl5.22, perl5.24, perl5.26, perl5.28, perl5.30, perl5.32, perl5.34, perl5.36 but the p5-foo ports have perl5.branches 5.28 5.30 5.32 5.34 . That can't be right

See #67830 for why there are no module ports for p5.36 and later yet. See commit history, ticket, and mailing list archives for why old perl versions were removed. For example 5.26 subports were removed in [96a5edbb2a667d1908886915cbd016bfdbc0a42d/macports-ports] at the same time that 5.32 subports were added. Granted that commit message does not say why but there was probably mailing list discussion approving the plan to remove old perl version subports beforehand (maybe not immediately before that commit, but before the practice of removing old perl version subports began). Another example, perl 5.16, 5.18, and 5.20 subports were removed in #50245.

(and frankly, that variable should be set in a PG and only overridden in individual Portfiles when unavoidable).

No. Portfiles are only indexed when they are modified. So, to add a new perl version subport to a port, it is not sufficient to change a portgroup that the port includes; *some* change also needs to be made to the Portfile, so it may as well be a change that relates: indicating which perl versions the subport is compatible with.

Also, I see the perl5 PG now misdefines perl5.major as major.minor, which is probably part of the reason of the mess.

As far as I know, the perl module ports work as intended, so I would hesitate to say that anything is misdefined. How is perl5.major defined, and how do you think it should be defined instead?

I do not see any evidence that perl is included in the port select mechanism.

Correct, it is not. The way the perl ports are set up in MacPorts predates the select mechanism and it's difficult to add it now, due to the longstanding practice of ports depending on port:perl5 and expecting it to provide /opt/local/bin/perl. If perl switches to the select mechanism, the perl5 port must be deleted and all ports that depend on it must switch to a specific perl version port, not just by declaring the dependency but also by adding a configure argument or other means of informing the build system which perl they must use. See #29763, #60812.

In other words, wouldn't it be much cleaner to provide a port:perl5 that is not just a dependent but installs an actual default version of perl5, which gets updated regularly but not more often than strictly necessary, and which can be used preferentially for things like build dependencies?

The pros and cons of switching to that method could be discussed. One con is that it be different from how we handle most other languages in MacPorts (python, php, ruby...).

Updating that main-default port would only have to trigger a rebuild of the corresponding p5-foo ports when there's been a ABI break (very rare, I assume since Linux distros do not seem to maintain separate lineages of perl packages).

Since we have not used this strategy so far, I do not know if that is true.

The opinion has been given several times before that offering separate perl versions in MacPorts is unnecessarily complicated and that we should go back to having just the one perl5 port (for the latest perl version) and just one p5-* port for each perl module. But it took considerable effort (even the invention of the subport feature of MacPorts!) to implement it in the first place and it would take considerable effort to de-implement it again, even if we agreed it was a good idea to do so.

This would address the issues I'm referring to above, and also put an end to (a priori) the ever-increasing amount of obsolete p5-foo ports that never get cleaned up if the user doesn't do it because "who knows what they're needed for".

The same could be said about python modules where versions that are EOL are regularly deleted. The user is expected to run sudo port reclaim (or just sudo port uninstall obsolete) periodically. MacPorts reminds the user to run reclaim periodically, unless the user has told MacPorts not to do that.

Of course it could also be an idea to allow advanced users to set a preferred perl5 version in macports.conf (which would get included in perl5.branches) - and let the perl5 PG print out a warning when it's really time to bump that version.

I don't think macports.conf at present contains any settings related to any specific port or portgroup and it's probably good to keep it that way.

[RJVB (René Bertin)]

Replying to ryandesign:

The title was meant to be provocative but I could have made it more clear that it's my opinion.

It has been this way since perl5 and the p5- ports were split into submodules in 2011 (#30638).

MacPorts has evolved considerably since then, so there must be a more convenient and cleaner way to do this nowadays.

Granted that commit message does not say why but there was probably mailing list discussion approving the plan

But who goes scavenging through mailing list archives to find such discussions?!

(and frankly, that variable should be set in a PG and only overridden in individual Portfiles when unavoidable).

No. Portfiles are only indexed when they are modified.

As a sidebar, that too is something that could end probably should be modified because it's counter-intuitive to anyone used to programming. But the revision mechanism exists for this purpose, which probably a whole lot simpler to use in a script than a more complex edit. Also, it is not impossible that the majority of ports do not need to be revbumped. As you say, perl5 is mature and rarely breaks ABI, very few perl ports install binaries (2 our of 146 for me; they do not link to libperl.dylib) and the rest simply don't need to be revbumped unless there are known incompatibilities.

Also, I see the perl5 PG now misdefines perl5.major as major.minor, which is probably part of the reason of the mess.

As far as I know, the perl module ports work as intended, so I would hesitate to say that anything is misdefined. How is perl5.major defined, and how do you think it should be defined instead?

perl5.major is currently defined as major.minor (so 5.28 in the current snapshot I have). Now I haven't checked if there's a Perl 6 but a priori the major version here is 5.

Correct, it is not.

So that's a good thing for the kind of change I was proposing.

The pros and cons of switching to that method could be discussed. One con is that it be different from how we handle most other languages in MacPorts (python, php, ruby...).

I cannot speak for the other two, but Python is different here in that binaries are much more common and they do link to a library from the main port. Plus the language is designed to have a version-specific "site-packages" or "dist" directory where packages are installed. Perl doesn't seem to have that (which is already a good indication that it isn't necessary).

But it took considerable effort (even the invention of the subport feature of MacPorts!) to implement it in the first place and it would take considerable effort to de-implement it again, even if we agreed it was a good idea to do so.

As I said, there's no need to de-implement it. Either the p5-* ports start depending on port:perl5 which provides some default version or there could be a port:perl5.x (x not being a number) with the corresponding p5.x-* ports. The former approach should be transparent for all ports that are now version-agnostic dependent on perl5.

The user is expected to run sudo port reclaim (or just sudo port uninstall obsolete) periodically.

I assume that gives very little control over what gets uninstalled, just like the -u option will uninstall all older (if not all other) versions of a port. Fine for Joe & Jane User but for more advanced users who always want to keep at least 1 older version around it's just not an option.

I don't think macports.conf at present contains any settings related to any specific port or portgroup and it's probably good to keep it that way.

You can set default variants in there (or an equivalent file), which can very well be specific to a single port or portgroup. Idem for the environment variables to preserve (I don't know if the feature is used that way or if it would be condoned). NB: the perl5 PG defines variants for perl5 versions. I don't see a clear documentation in the PG what this is supposed to do but using a version-specific variant would do the trick just as well (though more awkwardly) as using a version-defining variable.

[mascguy (Christopher Nielsen)]

I don't have much to add - Ryan covered every point, and provided a great amount of background - apart from posing a fundamental question:

• Are you in the process of updating any specific port(s) - and/or creating new ones - where you're encountering challenges, due to our current Perl 5 approach? If so, can you elaborate on those specific challenges?

Broader discussions are always good, to be sure. But it's also helpful to frame such discussion, relative to current work.