making port:xz optionally independent of other ports

[RJVB (René Bertin)]

There is an option in etc/macports/macports.conf to create xz'ipped software archives for ports built locally. That gives a substantial reduction in the disk footprint of those archives, but also introduces a dependency issue.

Xz depends on port:libiconv and port:gettext, and esp. the latter is updated rather frequently. The update sequence (when building from source) is as follows, taking gettext as an example (using xz compression and with the +universal variant which is the reason many of us have to build each upgrade from source):

1. port destroot gettext +universal
2. create the new archive in ${prefix}/var/macports/software/gettext/gettext???.txz
3. port deactivate gettext@current-version+universal
4. unpack the archive created in (2) in a temporary directory in ${prefix}/var/macports/software/gettext

And that's where things go wrong: since port:gettext has been deactivated, port:xz can no longer function.

The proposed enhancement allows to work around this by adding a default variant for xz, +nls. This default variant corresponds to the current situation, in which xz depends on libiconv and gettext to be "internationalised". Users who want to have xz compressed software archives can port install xz -nls, in which case said dependencies are dropped, and xz still functions when gettext or libiconv have been deactivated.

Upgrading xz itself will remain a problem.

The correct way to address this issue is of course to change the order of steps outline above:

1. port destroot xz +universal
2. create the new archive in ${prefix}/var/macports/software/xz/xz???.txz
3. unpack the archive created in (2) in a temporary directory in ${prefix}/var/macports/software/xz
4. port deactivate xz@current-version+universal
5. move the new files in place from the temporary directory into ${prefix}

This has 2 advantages: 1) the to-be-updated functionality remains available right up to the moment where standard system commands will be used to update it, so that tools used in the procedure can update themselves and 2) the "gap" during which the to-be-updated functionality is unavailable is reduced to the strict minimum, which means users are unlikely to notice it even when they're running the update in the background. This "gap" can be quite lengthy for large ports like Qt.

[larryv (Lawrence Velázquez)]

For what it’s worth, I don’t like this idea.

1. Variants are generally undesirable.
2. Frankly, we should not do anything to encourage people to use .txz archives until we make it reasonable to do so without the use of the xz port. As you already noted, systems that use .txz archives via the xz port would continue to break when xz itself is updated. The only stable way to address this would be to bundle xz with base.

[larryv (Lawrence Velázquez)]

Replying to rjvbertin@…:

The correct way to address this issue is of course to change the order of steps outline above:

I think this is a better idea than adding (IMO) frivolous variants to xz.

[RJVB (René Bertin)]

I resent that - frivolous would have been if I'd added an nls variant to a project that never had any need to talk anything else but the language it was conceived in O:-)

I agree of course that the outlined change in order would be a much better idea. I feel just a bit too uncomfortable with "base" to start looking into that myself, or else I'd already have done it.

[ryandesign (Ryan Carsten Schmidt)]

I agree that I don't want to add this variant and that I would rather work on getting an xz compressor/decompressor bundled with base. The hard work for that was already done when we started bundling tcl; I think adding more bundled software at this point would be easy. The question is which software.

The official xz has complicated licensing. From the COPYING file, "liblzma is in the public domain" and "xz, xzdec, and lzmadec command line tools are in the public domain unless GNU getopt_long had to be compiled and linked in from the lib directory. The getopt_long code is under GNU LGPLv2.1+." We would want to know whether the xz we bundle is public domain or LGPL-2.1+. Also, xz and liblzma are not multithreaded, as far as I know. We recently added support to MacPorts to use pbzip2, a parallel bzip2 implementation, which speeds up compression and decompression; it would be undesirable to now lose that speedup by moving to a different compression format.

There are a number of parallel xz implementations, which I think we should evaluate. There's ​pxz which I wanted to make a port of, but the official source download is ​over 5 years old and there has been no movement on my request to release a new version; also, it's GPL, but at least it's not GPL 3.

There's ​pixz which is 2-clause BSD licensed. It needs xz's liblzma but as per above that's public domain. I think I'll try to make a port for this. Its README points to two other parallel xz programs, plzip and ChopZip. The author seems to think that the indexable files pixz produces are better.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

There's ​pxz which I wanted to make a port of, but the official source download is ​over 5 years old and there has been no movement on my request to release a new version;

Forgot this was submitted in #46732.

There's ​pixz which is 2-clause BSD licensed. It needs xz's liblzma but as per above that's public domain. I think I'll try to make a port for this.

Done in r134404, but it also requires libarchive, which has its own dependencies.

[RJVB (René Bertin)]

Replying to ryandesign@…:

I agree that I don't want to add this variant and that I would rather work on getting an xz compressor/decompressor bundled with base. The hard work for that was already done when we started bundling tcl; I think adding more bundled software at this point would be easy. The question is which software.

Is there a reason not to change the order of operations as I outlined? That would (should...) remove the need to bundle more things with base.

I think that someone willing to figure out how to select xz for the software archives also won't have any problems installing port:xz or any of its alternatives.

[ryandesign (Ryan Carsten Schmidt)]

Replying to rjvbertin@…:

Is there a reason not to change the order of operations as I outlined? That would (should...) remove the need to bundle more things with base.

I'll tentatively say that sounds reasonable, but I don't know why the order of operations is the way it is, or how we would change it; I also have not fully investigated MacPorts base.

[larryv (Lawrence Velázquez)]

Replying to ryandesign@…:

The official xz has complicated licensing. From the COPYING file, "liblzma is in the public domain" and "xz, xzdec, and lzmadec command line tools are in the public domain unless GNU getopt_long had to be compiled and linked in from the lib directory. The getopt_long code is under GNU LGPLv2.1+." We would want to know whether the xz we bundle is public domain or LGPL-2.1+.

On my system, at least, xz does not use its bundled getopt_long.

% nm -m /opt/local/bin/xz | grep getopt
                 (undefined) external _getopt_long (from libSystem)
%

[mojca (Mojca Miklavec)]

Cc Me!

[larryv (Lawrence Velázquez)]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

I believe the xz-bootstrap port satisfies this request (an xz that has no dependencies).

Discussion of adding xz directly to base is #52000.