Add a buildbot with +universal for all packages

[mojca (Mojca Miklavec)]

It would be sooooooo great if packages like qt4-mac and compilers like clang would be built with +universal along the normal variants.

I would like to request either:

• to add a keyword to portfiles to signal to builtbot the desire to build a certain package and its dependencies with +universal (useful for qt4-mac, developers could add the flag to commonly used ports and those that require longest build times)
• or simply compile all packages with +universal

[jmroot (Joshua Root)]

This would be a MPAB feature, not a hosting change.

When initially implementing the buildbot, I thought about (more generally) specifying extra variant sets to build per-port, but I'd really like for that to be guided by statistics, not guesses.

In any case, it requires changes to both the input to MPAB and the status targets, to work with portname,variants rather than just portname.

[pixilla (Bradley Giesbrecht)]

Cc Me!

[rmstonecipher@…]

Cc Me!

[rmstonecipher@…]

Josh,
For me to build +universal wherever possible I only have to edit macports.conf and variants.conf.
Could a similar change be made to configuration files on the buildbot?

Ryan Stonecipher

[ryandesign (Ryan Carsten Schmidt)]

#37915 has the same request but for +quartz.

[larryv (Lawrence Velázquez)]

Replying to ryandesign@…:

#37915 has the same request but for +quartz.

Also +x11.

[rmstonecipher@…]

+universal would provide binaries that would run on any architecture supported by a darwin version regardless of user preferences.
+x11 and +quartz are (most of the time) either/or options based on user preferences.
Is #37915 really a duplicate, or a similar type of request with a distinctly different goal?

[cooljeanius (Eric Gallager)]

Cc Me!

[macports@…]

rmstonecipher: I didn't mean #37915 to be a duplicate, especially when this issue is worded as it is.

Anyway, any progress here?

[larryv (Lawrence Velázquez)]

Replying to macports@…:

Anyway, any progress here?

I do not believe anyone has been working on this.

[cooljeanius (Eric Gallager)]

This came up again on the mailing lists here: ​https://lists.macosforge.org/pipermail/macports-users/2013-May/032608.html (the port in question that triggered this coming up again was VirtualBox this time)

[cooljeanius (Eric Gallager)]

For cross-referencing purposes, #39768 wanted one for the +quartz variant...

Edit: looks like comment:6 above had another ticket (#37915) that was the same thing...

[mojca (Mojca Miklavec)]

Now that the build scripts have been reimplemented, deployed on a new hardware and with a new sysadmin), this finally sounds doable to me without the need for any additional virtual machines for the build slaves.

Ryan's idea was to automatically upload archives of +universal ports only when a package like wine asks for i386 or universal. In fact this is already working, so the ticket about "building +universal" could be closed as fixed as far as I'm concerned.

I'm not sure what the situation is with VirtualBox. Has i386 been "disabled" just because it was annoying to compile all the dependencies from source? If that's the case, the universal builds could probably be enabled again.

Some additional builds of +universal build tools (ld, clang++, ...) might be handy for older OSes (10.5 in particular; but we currently don't provide that one).

We could in principle build all the ports as +universal with a few additional lines of code (and by consuming approximately three times the resources we currently do), but maybe we should go with the current approach for now and see whether there is any need to add additional candidates for universal builds later on.

As far as +quartz is concerned: I started implementing preliminary support to build with additional variants. I will need some help extending the script that is currently responsible for installing dependencies (portname +quartz usually needs different dependencies than portname +x11), but once that gets fixed, it would enable manual builds with arbitrary variants for now (which we shouldn't start misusing).

Once that is in place, we could start implementing some heuristics to determine when to build a port with additional variants (either some hardcoded list of ports or all ports that provide a non-default +quartz variant etc.). The nice thing about the current buildbot setup is that we don't need any additional resources (other than some extra disk space and CPU clocks on existing infrastructure) to achieve that.

See also #51995.

[mojca (Mojca Miklavec)]

The main reason why I needed qt4-mac in the past was for example to satisfy the dependency of gnuplot +wxt +qt because wxWidgets 2.8 would not build as 64-bit and then this eventually pulled in qt4-mac +universal as a consequence. I don't know to what extent providing universal builds for all packages makes sense nowadays.

We are still affected by the bug that when wine's dependency gets updated that dependency would not be built as universal (and building wine again doesn't help because it has been built already). We could get the ability to manually build individual ports with +uniersal once #52742 gets deployed. It would probably still help to have some compilers built universally to allow universal builds of other software (manually). Other than that I'm not sure what we still want to implement/deploy.

[ryandesign (Ryan Carsten Schmidt)]

I don't think we should build and distribute binaries of all ports with the universal variant. It's not needed generally and would take a lot of disk space, and we've already lost one of our mirrors because they ran out of disk space. We also do not have unlimited disk space on the buildbot workers, and they keep (the current version of) all ports they've built installed.

We already build and distribute universal binaries when they are a dependency. So, when wine is updated, it builds universal variants of all its dependencies, such as freetype. All we need to do to complete that is whenever a port is updated, get the list of different variant combinations with which that port is currently installed on the buildbot worker, and schedule a build for each of those variant combinations. (So, if freetype is currently installed with the universal variant and with no variants, schedule two builds of freetype, one with the universal variant and one with no variants.)