Use hfscompression on buildbot

[ryandesign (Ryan Carsten Schmidt)]

Can we somehow use the new MacPorts base hfscompression functionality in the main /opt/local prefix on the buildbot workers? As I recall, the current implementation relies on the libarchive port being active, but on the buildbot, no ports are active in the main prefix. We could easily install a copy of libarchive in the separate tools prefix however. Is there a way we could allow that to be used by the main prefix?

[jmroot (Joshua Root)]

It should be used already on sufficiently recent OS versions. There has to be a bsdtar in PATH that supports the --hfsCompression flag.

[ryandesign (Ryan Carsten Schmidt)]

What would be sufficiently recent? Neither man bsdtar nor bsdtar --help mentions an --hfsCompression option, on any version of macOS.

The disk space problem is more acute on older versions of macOS, given how many different versions of clang need to be installed to build C++11 ports. So I'm looking for a solution that works on older systems as well.

What's the recommended way to determine whether files are already hfscompressed? I found a third-party afsctool that claims to be able to do it, and Apple's hfsdebug apparently can too.

[jmroot (Joshua Root)]

Hm, you're right, I guess Apple hasn't been updating their libarchive very quickly. I guess you just need to put a good bsdtar in binpath ahead of /usr/bin. You probably don't want to make everything in the tools prefix available since that will hide missing dependencies. Obviously you can still only use hfscompression on OS versions where the filesystem actually supports it, which is 10.6+.

[ryandesign (Ryan Carsten Schmidt)]

The MacPorts configure script has a --with-tar option. Will setting that to a bsdtar installed by the libarchive port (in a separate tools prefix) work to enable hfscompression support?

[jmroot (Joshua Root)]

No, the invocation that uses --hfsCompression finds bsdtar with binaryInPath. I think you need to install libarchive in the tools prefix, symlink bsdtar into some other directory, and add that directory to binpath just before /usr/bin.

source:macports-base/src/registry2.0/portimage.tcl#L354​

[ryandesign (Ryan Carsten Schmidt)]

That's what I thought. Why did we do it that way? It is very confusing. I don't want to modify bin_path on the buildbot workers because I am concerned that this will affect how some ports build.

I was going to ask why we didn't do it the way we handled parallel bzip2 support, but now that I've looked at it, we've handled that in a very confusing way as well. We allow the user to specify a bzip2 at configure time, which could be a parallel bzip2 such as lbzip2, but it looks like in many cases we ignore the user's request and choose a program from the path instead.

I feel that we should let the user specify these programs via configure args and if the user has done so we should always honor that request. If the user hasn't specified something, then we can choose what we think is best.

[ryandesign (Ryan Carsten Schmidt)]

Some success: In the tools prefix, I installed libarchive, then uninstalled inactive ports, deactivated all ports, activated libarchive, and activated inactive ports. This saved 3-4GB of space on the 10.6-10.9 workers. There was only negligible savings on the 10.10-10.14 workers because they don't have any clang ports installed.

So I imagine if I followed your suggestion and put a symlink of the tools prefix's libarchive's bsdtar in a directory by itself and added that directory to the main prefix's binpath, that would save at least 3-4GB of space when buildbot was building a port that required MacPorts clang, though incurring additional CPU time for all builds for compression and decompression. Maybe not keeping large and/or leaf ports installed would provide better savings without needing to incur that CPU penalty.

[ryandesign (Ryan Carsten Schmidt)]

In bd721c1228aab9f9d540be05376b3bb1fccaef46/macports-ports (master):

libarchive: Add zstd support and program symlinks

libarchive already used zstd opportunistically if it happened to be
installed at build time. This adds the dependency and makes explicit
that we want it.

The symlinks make it easier to use bsdtar from a libarchive installed by
one copy of MacPorts in another copy of MacPorts, such as how we want to
use it on the buildbot workers.

See: #57396

[ryandesign (Ryan Carsten Schmidt)]

In e70d3c9fc626744045c202397fa3ad77b8b1c040/mpbb (master):

Add --binpath option to mpbb-checkout

See: #57396

[ryandesign (Ryan Carsten Schmidt)]

In 2343a696d5897ecece48ef2212e9b4fb49b47923/macports-infrastructure (master):

Add --binpath when calling mpbb checkout

See: #57396

[ryandesign (Ryan Carsten Schmidt)]

The new buildmaster configuration has been deployed.

[cjones051073 (Chris Jones)]

Great, thanks.

Yesterday I committed an update to gcc8, bring it to 8.3.0. These builds failed on 10.7 and 10.8 due to disk space issues. Could you please trigger a rebuild now of this port (or is this something I can do ?) as it would be good to have the binary tarballs for these platforms ?

[mojca (Mojca Miklavec)]

I triggered the two builds. You can trigger those yourself if you have an account on buildbot. If you don't have one, you can ask Ryan for it (we should switch to oauth some time in the future).

[cjones051073 (Chris Jones)]

Thanks Mojca. I do occasionally wish to trigger builds. I normally do it via a revbu,p, or similar, but this is hardly ideal. So if Ryan is Ok with it an account on the builders might be useful.

[cjones051073 (Chris Jones)]

Was this pushed to all MP buildbots, or only some of the older ones ?

I ask as I am still seeing 'out of space failures', on the 10.8 buildbot.

​https://build.macports.org/builders/ports-10.8_x86_64_legacy-builder/builds/88009/steps/install-port/logs/stdio

[ryandesign (Ryan Carsten Schmidt)]

Yes, all machines have this configuration now. Observe in ​https://build.macports.org/builders/ports-10.8_x86_64_legacy-builder/builds/88009/steps/install-dependencies/logs/stdio how decompression occurred using bsdtar -xvp --hfsCompression.

HFS compression will only save a small amount of space. We need to pursue other strategies of reducing disk usage as well, such as #57464.