perl5: improve / reimplement packaging

[mojca (Mojca Miklavec)]

I'm opening this ticket to collect and vote on ideas with their pros and cons about different options to package:

• perl5[.x]
• perl modules p5[.x]-foo
• ports that depend on Perl
• perl6

Desired features:

• easy updates of perl modules, possibly (semi-)automated from metadata from CPAN
• painless upgrades from perl5.[n] to perl5.[n+1]
• reasonably small effort with modifying and revbumping all the ports that depend on Perl

Some particular aspects to think about:

• a port could have a flag (possibly "auto-generated") to indicate that there is no difference in what Perl version is being used (other than maybe path to perl, but path could be version-neutral); those ports won't need any revbumps after changing the version
• ...

Please try to think out of the box and and come up with proposals even if implementing them is not yet supported by the base. We can extend the base if we know exactly what we want to achieve.

Some relevant tickets that have been open for a long time already:

• #29763

  *_select framework does not apply to Perl: fix provided

  #36980

  perl5.x: sitebin & vendorbin flags may be inapproriate

  #43741

  cpan fails to install any module

(No need to rush into any decisions and implementations before we carefully think of all the consequences of that particular implementation.)

[RJVB (René Bertin)]

Some thoughts:

• ports that only require the perl interpreter (plus potentially perl scripts), be it as a build or runtime dependency should not have to include a perl PortGroup, ideally. Dependencies like port:perl5 (or port:perl6) and port:p5-uri should be enough.
• runtime dependencies of this sort should not require a revbump if perl gets updated, unless the update introduces an incompatibility or regression. I think the same applies to build dependencies of this sort (isn't perl often used to generate/translate documentation?) though the reproducible build principle may impose otherwise.
• to ease the revbumping burden (and reduce useless rebuilds), it might be good to agree on a way for ports to indicate that they don't care about the exact perl version (and thus can be skipped when revbumping).

• surely a brilliant MacPorts dev could find a way to remove the need to reinstall all those perl packages at each minor version upgrade? O:-)

[petrrr]

Cc Me!

[mojca (Mojca Miklavec)]

Here are some of my thoughts.

Disadvantages of current situation with many perl versions installed in parallel

• pointless effort in many ports that work with any version of Perl
• some programs get a -5.x suffix and cannot easily be found by other software that depends on them; a lot of special cases are needed
• almost no way to enforce consistent variants (if one port upgrades to perl5.y and its dependency stays at perl5.x, things break)
• very heterogeneous handling of ports w.r.t different perl versions
• we seem to need more than a year to do the transition now (and nearly forever if one doesn't force it)
• once users get +perl5_16 for one port, +perl5_18 for the other, ... it's very difficult to install/upgrade all ports to +perl5_22, so users get stuck on the "oldest" version forever
• (many others, too many to mention them all)

Disadvantages of having just one Perl interpreter available at any given time

• developer(s) will be "afraid" to upgrade perl5.x to perl5.(x+2)
• at the same time the person doing the upgrade has to:
  • upgrade Perl itself
  • revbump all ports that depend on perl (probably 2000+ of them)
  • deal with a large number of failures due to early adoption; even if we upgrade late, problems still won't show up until people actually do the upgrade
• inability to avoid failures on particular ports that will be incompatible with the latest version, with basically no grace period and zero testing

In comparison to linux distributions where most likely a fixed version of Perl is chosen and well tested before the official release, we have no "grace period", "stable vs. unstable" distribution or any easy way of testing or merging "massive changes".

Perl 5.14 was released in 2011. And it took us almost five years to do the jump just twice (5.12 -> 5.16 -> 5.22). When we were transitioning to 5.16, 5.16 was actually already outdated. Expecting from the maintainer of Perl 5 to do such a transition in an infinitely short period of time is a bit hard of a requirement.

My proposal

I would thus like to propose a solution that needs quite some extra coding in the core, but something that is a good compromise between the two extremes of providing just a single version of Perl and that of letting every single package and user choose which version of Perl to use.

My idea is the following:

• ban the idea of port select for perl (remove any traces of it) as it is not suitable/flexible enough for what we need and almost impossible to use
• perl5[.x]: Different versions of Perl are still supported and implemented in basically the same way as they are now. We could theoretically provide an arbitrary number of working versions of Perl (I don't see a compelling need to remove old Perls).
• p5-foo modules:
  • We put additional amount of work to simplify packaging (to be discussed separately).
  • But basically we would still allow parallel installations of p5.22-foo and p5.24-foo.
  • I would love it if there was no need to manually specify the complete list "5.20 5.22 5.24" etc. in every single port and if we could implement some trick in Portindex and buildbots to see and build the ports nonetheless when new perl versions are added. (Yes, this needs an extension to the code in core.) Failing to implement this point would just mean a tiny bit more work, but that work can be fully automated, so it's not that bad after all.
  • p5-foo would have multiple conflicting variants like +perl5_22 +perl5_24 with p5-foo +perl5_22 declaring dependency on p5.22-foo.
  • Other ports would declare dependency on p5-foo rather than on p5.22-foo.
• Ports that just use Perl and don't link against perl's dylib or install modules under ${prefix}/lib/perl5/vendor_perl/5.x/:
  • These ports would depend on perl5 and p5-foo.
  • Their scripts would be executed with ${prefix}/bin/perl (or maybe ${prefix}/bin/perl5 just to be on the safe side :)
  • After an upgrade of Perl (the official switch of the default version), there will be no need to revbump the port or do anything else with it. It should just keep working.
  • (These ports could still include the perl5 PortGroup if necessary.)
• Ports that link against libperl.dylib or install files under ${prefix}/lib/perl5/vendor_perl/5.x/ or have some part of perl hardcoded in some way:
  • These ports should include "PortGroup perl5 1.x" and "perl5.something_to_create_variants" in the Portfile.
  • When default version of Perl changes, these ports won't need a revbump either because users would replace foo +perl5_x with foo +perl5_(x+2).
  • But the ports will need a rebuild on the buildbots to get binary packages with the new default variant.
• Switching from one version of perl to another:
  • The macports base should enforce the use of consistent variants across the complete installation (in a similar way as it enforces +universal on all dependencies).
  • We should implement a special command that does the switch.
    • The command would go through the list of all ports with an active +perl5_x variant and reinstall all of the ports with +perl5_y.
    • p5.x-foo modules would remain installed (they would not be removed or replaced_by until the graveyard replaces them with the latest version).
    • p5-foo +perl5_x would be replaced with p5-foo +perl5_y, installing p5.y-foo if necessary.
    • All other foo +perl5_x would be replaced by foo +perl5_y in a proper order to make sure that dependencies get changed first.
    • Some special safeguards need to be implemented for cases when the transition gets interrupted in the middle of the process.
• At any given time we would basically keep just the latest two maintained versions (or one of them):
  • Brave developers would be free to try perl5.(2n+1) on their own, without having to do an unreasonable amount of work to get development releases working.
  • As soon as a new stable release comes out, the perl5 maintainer prepares a new port (say, perl5.24), increases the list of available default variants, adds subports for perl modules p5.24-foo and variants to ports that need them. At that point regular users don't get automatically upgraded and basically don't notice the change, but brave developers can easily switch to that new version.
  • Sufficient time (several months up to a year?) to fix problems.
  • Once brave developers & users reach a consensus that the new version of Perl is usable and stable enough (probably a couple of months after the new release is out), the global default in MacPorts is changed. I'm not sure how to enfornce that change to users (they could get a warning/note that they should issue a particular command), but users can still switch back and forth between the two versions as they please.
  • Once the consensus is reached that the new version works fine, or around the time when the old version is no longer maintained, we remove the subports and variants for that version, again ending up with either a single version or possibly replacing the old version with the new one. Users that sticked with that version should get an automatic upgrade at that point at latest.

What we need:

• A more advanced tool to help port maintainers *automatically* determine:
  • The list of dependencies (tracemode should be able to provide that list).
  • Whether a port is noarch.
  • Whether a port links against perl5.dylib or installs versions-specific files/modules for Perl somewhere.
• Both more advanced and more lightweight basic functionality of the perl5 PortGroup, so that maintainers won't have to think twice about consequences before including the PortGroup in their ports.
  • Automatic generation of variants +perl5_x (with one rather than five lines of code).
  • A better name for ${perl5.major} :)
• Buildbot:
  • A doable way to rebuild all ports with the new +perl5_x variant.
• Portindex:
  • Ability to reindex ports containing "perl5.something_to_create_variants" when the set of available variants increases.
• Graveyard
  • More efficient build with portindex and more efficient failures on the buildbot.
• ...

[dliessi (Davide Liessi)]

Cc Me!

[mojca (Mojca Miklavec)]

See also #51498.

[danielluke (Daniel J. Luke)]

That is a lot of effort (for multiple simultaneous perl version installs) seemingly mostly because we're afraid of breaking things on upgrade (I don't think there is other benefit from keeping unsupported versions around). Adding 'automatically' added versions to p5- ports also adds a bunch of (possibly broken) ports that will likely not be tested by maintainers.

I think we would be better off working on automation to run p5 port test suites to make it easier to validate ports and fix (or abandon) them when we want to move to the current perl5 release.

[RJVB (René Bertin)]

I'm all for making the upgrade process as automatic as possible. One can end up with a lot of automatically installed p5* ports that are dependencies of other ports, and which the casual user won't be aware of. I know I already abhor having to update all those packages/ports (getting spamassassin to work again etc), and clean up those that are no longer required. I can only (but would rather not) dream of what that would be like for users for whom "perl" is just a misspelled bauble in search of its mother ;)

adds a bunch of (possibly broken) ports that will likely not be tested by maintainers.

I'd look at it this way: they're likely to be reported and thus tested if they become broken, and maintainers will take that bridge when they come to it. I'd not advocate that kind of approach for standalone ports. For ports that are generated automatically I don't really see a reason to start looking to fix things that may not be broken if the benefits of the automatisation are big enough.

That doesn't mean that an automated test procedure wouldn't be a good thing to have.

[mojca (Mojca Miklavec)]

Replying to dluke@…:

That is a lot of effort (for multiple simultaneous perl version installs) seemingly mostly because we're afraid of breaking things on upgrade

The fact is that a whole lot of ports depend on working perl. You don't notice it now because things keep working. A trivial problem with the new version of Perl could leave a great portion of MacPorts broken.

(I don't think there is other benefit from keeping unsupported versions around).

We are not talking about perl5.8, but about two versions at most (at the moment that would be perl5.22 and perl5.24).

Adding 'automatically' added versions to p5- ports also adds a bunch of (possibly broken) ports that will likely not be tested by maintainers.

You mean mostly tested by nomaintainer? :) There are soooooooo many perl (sub)ports that it's literally impossible to wait for maintainers to upgrade them individually. And unless we provide means to easily test a new version of Perl, even those willing to do the testing won't be able to (I'm unable to test a few perl modules I maintain without updating tens of other modules first; we would end up in a deadlock). In any case I'm sure that switching to a single version without adding more functionality to the base (or to some external tools like buildbot) would be even more devastating than it is now.

You probably do realise that if we will in fact support just a single version of perl, those "automatically added" will just become "automatically revbumped" and receive equally "zero" testing from maintainers?

In case of changes like r148570 (which I admit should be done with more care on my side and I'm sorry for that), you would get a notice a few days in advance that things are going to change, but if perl5.26 was incompatible with your port, the revbump would actually break the functionality of the port (rather than just add a broken submodule that wouldn't be used by anyone). We could wait for a while, but I'm sure we wouldn't be able to wait for each and every of the thousands of ports to be tested. And without testing we wouldn't be able to report problems upstream.

I think we would be better off working on automation to run p5 port test suites to make it easier to validate ports and fix (or abandon) them

Writing tools to simplify testing would certainly be of enormous added value. This is one point where I fully agree. (Someone just needs to start contributing the code ...)

[mojca (Mojca Miklavec)]

Just one (un)related quotation from IRC:

forget search.cpan.org
no one knows how it works
we don't have access to the code or servers, as I understand it
use ​metacpan.org for all the things

This was in response to me noticing that ​http://search.cpan.org/dist/Attribute-Handlers/ lists 0.96 as the latest version, while ​http://api.metacpan.org/release/Attribute-Handlers says it's 0.99.

So if we do change things, we should better rely on metacpan ...

Also: ​https://github.com/gitpan/

[danielluke (Daniel J. Luke)]

Replying to mojca@…:

The fact is that a whole lot of ports depend on working perl. You don't notice it now because things keep working. A trivial problem with the new version of Perl could leave a great portion of MacPorts broken.

I don't see how having multiple versions of perl around makes this better.

We are not talking about perl5.8, but about two versions at most (at the moment that would be perl5.22 and perl5.24).

It seems really silly to do this just because we can't test / someone can't just upgrade a local perl, revbump modules, and fire off a script to validate them/find broken ones.

Adding 'automatically' added versions to p5- ports also adds a bunch of (possibly broken) ports that will likely not be tested by maintainers.

You mean mostly tested by nomaintainer? :)

I'm just pointing out that the proposal keeps multiple perls around because it's too hard to test while at the same time introducing an automatic change that creates a bunch of untested ports.

There are soooooooo many perl (sub)ports that it's literally impossible to wait for maintainers to upgrade them individually.

We really should be autogenerating Portfiles for all of CPAN (or hacking in support in base for cpan-managed installs for all perl modules).

And unless we provide means to easily test a new version of Perl, even those willing to do the testing won't be able to (I'm unable to test a few perl modules I maintain without updating tens of other modules first; we would end up in a deadlock).

If the testing is automated, one developer can identify problematic ports.

If there's more than one developer who is actually going to work on it - a private branch (or private "local" repo) would work well.

In any case I'm sure that switching to a single version without adding more functionality to the base (or to some external tools like buildbot) would be even more devastating than it is now.

It simplifies things, which I believe makes solving the remaining problems more tractable. It's true that it doesn't fix all of the problems we have with perl (although it does fix a bunch of them).

And without testing we wouldn't be able to report problems upstream.

one 'supported' perl version doesn't equal no testing.

two 'supported' perl versions doesn't equal testing.

I think we would be better off working on automation to run p5 port test suites to make it easier to validate ports and fix (or abandon) them

Writing tools to simplify testing would certainly be of enormous added value. This is one point where I fully agree. (Someone just needs to start contributing the code ...)

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca@…:

We are not talking about perl5.8, but about two versions at most (at the moment that would be perl5.22 and perl5.24).

But removing old perl versions (i.e. removing old perl version modules and old perl version variants) breaks user systems in confusing ways as reported in #51498. If we want to make a habit of removing old perl versions we'd want a solution for that situation.

[danielluke (Daniel J. Luke)]

Replying to ryandesign@…:

But removing old perl versions (i.e. removing old perl version modules and old perl version variants) breaks user systems in confusing ways as reported in #51498. If we want to make a habit of removing old perl versions we'd want a solution for that situation.

yet another issue that is fixed in the future if we just have one perl version and one set of perl modules.

[RJVB (René Bertin)]

Isn't that a chicken and egg situation unless you plan to remain on the same single perl version forever? (Now if that could happen ...)

[danielluke (Daniel J. Luke)]

Replying to rjvbertin@…:

Isn't that a chicken and egg situation unless you plan to remain on the same single perl version forever? (Now if that could happen ...)

no.

It's one transition from multiple perl5 ports and p5- ports to a single set. When a new perl is released the single perl5 port is upgraded an the p5- ports are upgraded/revbumped. A user running 'port upgrade outdated` will get a new/updated working perl install.

[RJVB (René Bertin)]

As long as the perl5 and p5-foo ports remain around I presume nothing much will change except for ports that

• depend on binary content in a p5- port - those ports will probably need more attention by their maintainer
• ports that for some reason don't work with a newer perl version. I'm not sure what you'd do with those anyway.

However, I'd strongly suggest to develop something (a script, to be installed by the main perl5 port?) which allows to update/upgrade only the perl install. Regular users should probably still keep running the usual port selfupdate && port upgrade outdated sequence, but there should be something for advanced users who live on the edge too. Maybe that something would be as simple as a port upgrade perl5 followed by a loop that upgrades all installed p5-* ports, but ideally there would also be some logic in place to fetch new Portfiles selectively. This could also be a specific port; that would allow the perl port devs to push tested versions of the updating script, when they're ready, either through an upgrade of that port, or through an independent download.

In light with those last thoughts: selective updating would probably be easier if all perl portdirs were moved into a dedicated subdirectory so one could do something like (cd $(port dir perl5)/.. ; svn up).

Another approach I'm testing with my own KF5 ports: specify the current version (and possibly a "latest" version) in the PortGroup which all ports providing perl-related things are supposed to include anyway (unless that messes up the configure.cmd, build.cmd etc, like the Python PortGroup does?). It seems to work, though I have occasional ports that don't pick up a version upgrade after running portindex.

Mojca also asked for out of the box thinking: what if the p5- ports simply tapped into existing CPAN functionality?