automake @1.17_0 fails to install on 10.4

[fhgwright (Fred Wright)]

This is solely an issue with installing the precompiled binary; it builds successfully from source. Precompiled binaries don't usually exist for 10.4, but this port has a wildcarded OS version.

Partial error output is:

...
MacT @15:40:37.937: --->  Fetching archive for automake
MacT @15:40:40.058: --->  Attempting to fetch automake-1.17_0.any_any.noarch.tbz2 from http://bos.us.packages.macports.org/automake
MacT @15:40:46.148: --->  Attempting to fetch automake-1.17_0.any_any.noarch.tbz2.rmd160 from http://bos.us.packages.macports.org/automake
MacT @15:40:50.445: --->  Installing automake @1.17_0
MacT @15:40:50.632:: /usr/bin/tar: Skipping to next header
MacT @15:40:50.632:: /usr/bin/tar: Skipping to next header
MacT @15:40:50.632:: /usr/bin/tar: Skipping to next header
MacT @15:40:50.633:: /usr/bin/tar: Skipping to next header
MacT @15:40:50.633:: /usr/bin/tar: Skipping to next header
MacT @15:40:50.633:: /usr/bin/tar: Archive contains obsolescent base-64 headers
MacT @15:40:50.635:: /usr/bin/tar: Skipping to next header
...

It appears to be an issue with /usr/bin/tar, which is GNU tar 1.14 on 10.4, and GNU tar 1.15.1 on 10.5. It's probably fixable by using the MacPorts gnutar on 10.4, though this would be a "binary extract dependency" rather than an "extract dependency". Alternatively, perhaps the binary archive could be constructed in a more compatible fashion.

It's not clear that this issue is limited to automake; this may simply be the first "any" port that's been updated since that mechanism was created (or the first that's been updated since a change in how binary archives are constructed).

[ryandesign (Ryan Carsten Schmidt)]

Since this has to do with binary archives, which are not under individual ports' control but are created at the direction of MacPorts base, this would be a base bug and not specific to automake.

I'm not aware of any change in how our binary archives are constructed. It may always have been the case that some or all archives created on newer macOS versions cannot be extracted on 10.4's version of tar, but that this has only become a problem since it has become possible for archives to be shared amongst OS versions (MacPorts 2.9.0?). I don't know which version of macOS created the automake 1.17 archive now on our servers.

You could try creating any tar archive on macOS 14 or other newer OS versions and then try extracting it on 10.4. If it fails the same way, and if you can then find some flags or options that can be used to create an archive on newer macOS that can be extracted on older macOS, base can be updated to use those flags or options.

[fhgwright (Fred Wright)]

Replying to ryandesign:

Since this has to do with binary archives, which are not under individual ports' control but are created at the direction of MacPorts base, this would be a base bug and not specific to automake.

As I suspected. That suggests making the summary more generic, which I don't have permissions to do.

I'm not aware of any change in how our binary archives are constructed. It may always have been the case that some or all archives created on newer macOS versions cannot be extracted on 10.4's version of tar, but that this has only become a problem since it has become possible for archives to be shared amongst OS versions (MacPorts 2.9.0?). I don't know which version of macOS created the automake 1.17 archive now on our servers.

Given that this issue just appeared between last Saturday and a little over a week earlier, it's probably whatever the buildbots are currently using.

You could try creating any tar archive on macOS 14 or other newer OS versions and then try extracting it on 10.4. If it fails the same way, and if you can then find some flags or options that can be used to create an archive on newer macOS that can be extracted on older macOS, base can be updated to use those flags or options.

If there's a read-side fix, that would be better, since it would be compatible with the existing archives. Though a write-side fix might be accompanied by a sweep to update the existing relevant archives (perhaps just by repackaging the existing tarballs, rather than rebuilding them).

The immediate workaround is for users to set buildfromsource on 10.4 to always (which I've confirmed works). If the problem can't be fixed soon, then it might make sense for base to make that the default on 10.4. That could even be done right away, to be reverted if and when a better fix comes along.

It would also be helpful, albeit in a rather verbose way (and the error output from this is really verbose), to have a failure to extract a binary archive fall back to building from source, just as it would do if a suitable binary archive didn't exist.

[ryandesign (Ryan Carsten Schmidt)]

Replying to fhgwright:

I'm not aware of any change in how our binary archives are constructed. It may always have been the case that some or all archives created on newer macOS versions cannot be extracted on 10.4's version of tar, but that this has only become a problem since it has become possible for archives to be shared amongst OS versions (MacPorts 2.9.0?). I don't know which version of macOS created the automake 1.17 archive now on our servers.

Given that this issue just appeared between last Saturday and a little over a week earlier, it's probably whatever the buildbots are currently using.

The point is, we have over a dozen different machines running macOS versions between 10.6 and 14, and I don't know which one of those created the current automake 1.17 archive. It looks like most of the machines attempted the build, probably because there were no other builds running at the time that the commit came through, but I don't know which of them finished first or last, and I'm not sure whether the one that finishes first or the one that finishes last is the one that ends up on the server.

You could try creating any tar archive on macOS 14 or other newer OS versions and then try extracting it on 10.4. If it fails the same way, and if you can then find some flags or options that can be used to create an archive on newer macOS that can be extracted on older macOS, base can be updated to use those flags or options.

If there's a read-side fix, that would be better, since it would be compatible with the existing archives. Though a write-side fix might be accompanied by a sweep to update the existing relevant archives (perhaps just by repackaging the existing tarballs, rather than rebuilding them).

Sure, if you find a way to fix the problem at extract time, let us know that. It's obviously not as simple as just making all ports depend on MacPorts gnutar for extraction since that would introduce a dependency cycle for MacPorts gnutar and everything it depends on.

To rebuild archives with an OS version whose tar version is compatible with Tiger, we would first have to be able to identify which archives are affected. If you can provide a method to do that, let us know.

[berg-michael (Michael Berg)]

I am not able to prove this definitively as I don't have a recent macOS machine to test.

However, I think this might be caused by recent versions of bsdtar which default to the pax archive format. I see rumblings about these changes (​https://www.undeadly.org/cgi?action=article;sid=20240417053301, ​https://bugs.python.org/issue36268) in other environments and suspect it happened on macOS as well.

Nominally, gnutar 1.14 should be able to handle this. Actually, it does successfully extract the archive with no errors. It only produces an error when asked for a specific file in such an archive, and it's still able to extract the requested file first. It seems like there is just some bug in this old version of gnutar.

My evidence:

py310-setuptools 72.2.0 used to be broken in the same way on 10.4. 74.1.2 works fine.

If one runs

bzip2 -cd py310-setuptools-72.2.0_0.darwin_any.noarch.tbz2  | less -

or

bzip2 -cd automake-1.17_0.any_any.noarch.tbz2  | less - 

one will find ./PaxHeader in the output. However, running

bzip2 -cd py310-setuptools-74.1.2_0.darwin_any.noarch.tbz2  | less -

i.e. the same test on a known working file, there is no ./PaxHeader.

(there's probably a more robust way to determine the format, but this was good enough for me).

I suspect there will not be a great way to work around this at extract time. gnutar is able to extract the file successfully, as I mentioned, but not if a specific file is requested.

Probably, the best solution is to ensure that tarballs are generated in the GNU format going forward. I believe this was the prior default, though I am open to being corrected. Virtually everything I know about tarballs was learned in the last few hours or so.

It should be possible to do a sweep of the existing tarballs and re-tarball them in the older format. This would not require rebuilding, just repackaging. The scope of the sweep probably could be narrowed by not considering files with a creation date before the default was changed to Pax, if such a date can be determined.

[ryandesign (Ryan Carsten Schmidt)]

Before we develop a script to identify and repack all affected archives, MacPorts base should be changed to ensure that new archives are made in the old format even on new versions of macOS. That's assuming there is a way to do so.

[fhgwright (Fred Wright)]

Replying to berg-michael:

I am not able to prove this definitively as I don't have a recent macOS machine to test.

However, I think this might be caused by recent versions of bsdtar which default to the pax archive format. I see rumblings about these changes (​https://www.undeadly.org/cgi?action=article;sid=20240417053301, ​https://bugs.python.org/issue36268) in other environments and suspect it happened on macOS as well.

[...]

I don't think that's the issue, for a couple of reasons:

1. AFAIK the extract phase normally unpacks entire tarballs, not individual files.

2. The error message (excerpted above) doesn't sound like it relates to this issue.

It should be possible to do a sweep of the existing tarballs and re-tarball them in the older format. This would not require rebuilding, just repackaging. The scope of the sweep probably could be narrowed by not considering files with a creation date before the default was changed to Pax, if such a date can be determined.

Replying to ryandesign:

Before we develop a script to identify and repack all affected archives, MacPorts base should be changed to ensure that new archives are made in the old format even on new versions of macOS. That's assuming there is a way to do so.

Indeed.

But something that could be done right away (as I suggested above) and hasn't been (unless it was done without referencing this ticket) is to make always the default for build_from_source on 10.4. Then, if there's no better fix before the 2.10.2 release, it will at least have the workaround.

On a somewhat related note, it doesn't appear that the buildbot framework has been updated to optimally handle the darwin any case. Such ports still get unnecessarily built for all platforms. I don't know what determines which result winds up on the archive servers. In the most simple-minded treatment, it would probably be all of them, with the last one "winning".

[berg-michael (Michael Berg)]

Replying to fhgwright:

Replying to berg-michael:

I am not able to prove this definitively as I don't have a recent macOS machine to test.

However, I think this might be caused by recent versions of bsdtar which default to the pax archive format. I see rumblings about these changes (​https://www.undeadly.org/cgi?action=article;sid=20240417053301, ​https://bugs.python.org/issue36268) in other environments and suspect it happened on macOS as well.

[...]

I don't think that's the issue, for a couple of reasons:

1. AFAIK the extract phase normally unpacks entire tarballs, not individual files.

Things are getting hung up on this command:

"exec -ignorestderr [findBinary tar ${portutil::autoconf::tar_path}] -xOj${qflag}f $archive_location ./+CONTENTS"

which evaluates to

tar -xOjf automake-1.17_0.any_any.noarch.tbz2 ./+CONTENTS

The intended output of this command AIUI is simply the contents of ./+CONTENTS to stdout. And we do see that output logged. If I drop the ./+CONTENTS gnutar unpacks the whole tarball without any complaints.

2. The error message (excerpted above) doesn't sound like it relates to this issue.

I agree, with the caveat that I think this version of gnutar is probably just incapable of telling us what's going wrong.

Further investigation:

gnutar 1.34 on Ubuntu also dislikes the file and gives a more helpful error (truncated):

tar -xOvjf "automake-1.17_0.any_any.noarch.tbz2" ./+CONTENTS
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
./+CONTENTS
@name automake-1.17_0
@portname automake
@portepoch 0
@portversion 1.17
@portrevision 0
@archs noarch
opt/local/bin/aclocal
@comment MD5:733b7a1979a407778c4970c48941270c
@comment binary:0
opt/local/bin/aclocal-1.17
@comment MD5:733b7a1979a407778c4970c48941270c

[...]

tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'
tar: Ignoring unknown extended header keyword 'SCHILY.fflags'

This lead me to a semi-informative discussion here ​https://github.com/nodejs/node/issues/22805, which included the following comment:

FWIW I've encountered some systems with older versions of (gnu) tar that will actually exit with a non-zero exit code and produce error messages when extracting the tarballs, despite the contents being extracted seemingly correctly. So I'd classify it as more than an annoyance.

The proposed solutions to this problem seem to be:

0. Suppress the warning at extract-time with --warning=no-unknown-keyword. This doesn't exist on gnutar 1.14 so is not an option.

-or-

1. Instead of using macOS bsdtar, use gnutar

-or-

2. In macOS bsdtar, use --no-xattrs at archive-time

-or-

3. In macOS bsdtar, use --format gnutar or --format ustar at archive time.

-or-

4. Some amalgam of these options.

I like option 2 the most, followed by option 3. But I don't know if they work.

If someone with a recent macOS machine (ideally 14 as I think that's most likely to be the culprit) could try the following and send the files I'd like to see how Tiger handles each output.

Note that if one of these works, it implies that it should be possible to change base to that configuration before repacking affected archives.

#!/bin/bash

# URL and file names
url="https://packages.macports.org/automake/automake-1.17_0.any_any.noarch.tbz2"
downloaded_file="automake-1.17_0.any_any.noarch.tbz2"

# Download the file
curl -O "$url"

# Extract the downloaded archive
mkdir -p extracted
bsdtar -xvf "$downloaded_file" -C extracted

# 1. Re-archive using --format ustar
bsdtar -cvjf archive_ustar.tbz2 --format ustar -C extracted .

# 2. Re-archive using --format gnutar
bsdtar -cvjf archive_gnutar.tbz2 --format gnutar -C extracted .

# 3. Re-archive using --format pax
bsdtar -cvjf archive_pax.tbz2 --format pax -C extracted .

# 4. Re-archive using --format ustar --no-xattrs
bsdtar -cvjf archive_ustar_no_xattrs.tbz2 --format ustar --no-xattrs -C extracted .

# 5. Re-archive using --format gnutar --no-xattrs
bsdtar -cvjf archive_gnutar_no_xattrs.tbz2 --format gnutar --no-xattrs -C extracted .

# 6. Re-archive using --format pax --no-xattrs
bsdtar -cvjf archive_pax_no_xattrs.tbz2 --format pax --no-xattrs -C extracted .

# 7. Re-archive using --no-xattrs
bsdtar -cvjf archive_no_xattrs.tbz2 --no-xattrs -C extracted .

# 8. Re-archive with no special arguments
bsdtar -cvjf archive_default.tbz2 -C extracted .

# Clean up
rm -rf extracted
rm "$downloaded_file"

echo "Re-archiving and cleanup completed."

[berg-michael (Michael Berg)]

I tried my script above to produce test files on bsdtar 3.5.2/Catalina. I tested whether they could be extracted on 10.4. My findings indicate that the pax archive format - which is the default for newer macOS - is the culprit.

In the text below, "doesn't work" means gnutar produced the usual Skipping to next header error on 10.4

"works" means gnutar extracted the file as expected and exited normally.

tar -xOvjf archive_default.tbz2 ./+CONTENTS - doesn't work

tar -xOvjf archive_pax.tbz2  ./+CONTENTS - doesn't work

tar -xOvjf archive_no_xattrs.tbz2 ./+CONTENTS - doesn't work

tar -xOvjf archive_pax_no_xattrs.tbz2 ./+CONTENTS - doesn't work

tar -xOvjf archive_gnutar.tbz2 ./+CONTENTS - works

tar -xOvjf archive_gnutar_no_xattrs.tbz2 ./+CONTENTS - works

tar -xOvjf archive_ustar.tbz2 ./+CONTENTS - works

tar -xOvjf archive_ustar_no_xattrs.tbz2 ./+CONTENTS - works

This implies to me that in base we should pass either --format ustar or --format gnutar when producing archives. --no-xattrs seems to have no effect so there's no reason to include it.

[ryandesign (Ryan Carsten Schmidt)]

Replying to fhgwright:

On a somewhat related note, it doesn't appear that the buildbot framework has been updated to optimally handle the darwin any case. Such ports still get unnecessarily built for all platforms. I don't know what determines which result winds up on the archive servers. In the most simple-minded treatment, it would probably be all of them, with the last one "winning".

Feel free to file a new ticket where we can discuss this.