Opened 8 years ago

Last modified 2 years ago

#52742 assigned enhancement

buildbot: add support for building ports with variants

Reported by: mojca (Mojca Miklavec) Owned by: neverpanic (Clemens Lang)
Priority: Normal Milestone:
Component: buildbot/mpbb Version:
Keywords: buildbot Cc: larryv (Lawrence Velázquez), raimue (Rainer Müller), neverpanic (Clemens Lang), ryandesign (Ryan Carsten Schmidt), ctreleaven (Craig Treleaven), danchr (Dan Villiom Podlaski Christiansen), yan12125 (Chih-Hsuan Yen), l2dy (Zero King), dliessi (Davide Liessi)
Port:

Description

It would be very handy to have some limited support for building with variants.

Before trashing the subversion checkout on my buildbot, I wanted to attach partial patches.

See #51995.

Attachments (3)

variants-master.cfg.diff (2.4 KB) - added by mojca (Mojca Miklavec) 8 years ago.
patches for master.cfg that add support for building with arbitrary variants (no sanity checking yet!)
variants-mpbb-fully-outdated.diff (2.6 KB) - added by mojca (Mojca Miklavec) 8 years ago.
outdated patch for basic support of variants in mpbb
variants-master.cfg.2.diff (2.4 KB) - added by mojca (Mojca Miklavec) 8 years ago.
add a field to support variants

Download all attachments as: .zip

Change History (23)

Changed 8 years ago by mojca (Mojca Miklavec)

Attachment: variants-master.cfg.diff added

patches for master.cfg that add support for building with arbitrary variants (no sanity checking yet!)

Changed 8 years ago by mojca (Mojca Miklavec)

outdated patch for basic support of variants in mpbb

comment:1 Changed 8 years ago by mojca (Mojca Miklavec)

The patches are pretty outdated, but they should give the first impression. I wanted to fix the second one, but I should probably write some code to parse options between install-port and <portname> first.

Some hints by Clemens about how to proceed:

tools/dependencies.tcl just opens all ports without any specified variants. See tools/dependencies.tcl line 66; the last parameter to mportopen, which is currently {}, is the list of variants. The format is easy to create:

array set variations {}
set variations(quartz) "+"
set variations(x11) "-"
mportopen ... [array get variations]

gives you an mport reference as if the port was installed with +quartz-x11. You should be able to find some suitable parsing code for variants given on the command line in src/port/port.tcl, line 1695 and following.

Looking at that file I have an impression that we forgot to consider global settings ([array get global_variations]):

if {!$index_only} {
    # Add any global_variations to the variations
    # specified for the port (so we get e.g. dependencies right)
    array unset merged_variations
    array set merged_variations [array get variations]
    foreach { variation value } [array get global_variations] {
        if { ![info exists merged_variations($variation)] } {
            set merged_variations($variation) $value
        }
    }

This means that even if someone would (theoretically) set up a buildbot with +quartz -x11 in configuration file, those setting would probably be ignored. Or do I misunderstand the concept?

Yes, I think the file doesn't handle global variants. I think it's also bad design that port/port.tcl has to do the merging of variant specs without at least some library support from macports1.0, but that's a story for a different refactoring.

An additional warning:

However, this output will only list positive variants, it will not list default variants that have been explicitly disabled. This might be a problem, because specifying -x11 +quartz will just give you a list of all dependencies with +quartz, but not without +x11, so if a port is written in a way that it does not automatically disable x11 if +quartz is given (for example if both are supported simultaneously and x11 is the default), this might lead to unexpected results. I haven't tested any of this, but we probably should.

comment:2 Changed 8 years ago by mojca (Mojca Miklavec)

Summary: buildbot: add support for building variantsbuildbot: add support for building ports with variants

comment:3 Changed 8 years ago by mojca (Mojca Miklavec)

It looks like the core of the needed functionality was already added in r152489.

comment:4 Changed 8 years ago by jmroot (Joshua Root)

Some of the logging probably needs to be enhanced with variant info, but that's about it AFAICT.

comment:5 Changed 8 years ago by mojca (Mojca Miklavec)

I didn't check the logs, but among other things we also need:

  • (straightforward/trivial) a revised version of my old attempt to add a field for variants to master.cfg
  • (lower priority) some basic logic to automatically build +quartz for certain ports

comment:6 Changed 8 years ago by mojca (Mojca Miklavec)

Keywords: buildbot added
Port: buildbot removed

Changed 8 years ago by mojca (Mojca Miklavec)

Attachment: variants-master.cfg.2.diff added

add a field to support variants

comment:7 Changed 8 years ago by mojca (Mojca Miklavec)

The new (trivial) patch for master.cfg seems to work for me.

We need some basic safeguard (regexp?) though to make sure that only valid variant names are entered, otherwise people could end up writing port names or any other weird things. (I could imagine things equivalent to sql injections ending up here. Nothing to do with sql, but I hope you get the idea.)

This will be sufficient to get gkt3 +quartz and other binaries on the buildbots. The next (independent) step would be to write the logic to build that automatically for some ports.

comment:8 Changed 8 years ago by ctreleaven (Craig Treleaven)

Cc: ctreleaven added

comment:9 Changed 7 years ago by danchr (Dan Villiom Podlaski Christiansen)

Cc: danchr added

comment:10 Changed 7 years ago by yan12125 (Chih-Hsuan Yen)

Cc: yan12125 added

comment:11 Changed 7 years ago by l2dy (Zero King)

Cc: l2dy added

comment:12 Changed 7 years ago by dliessi (Davide Liessi)

Cc: dliessi added

comment:13 Changed 7 years ago by neverpanic (Clemens Lang)

Component: server/hostingbuildbot/mpbb

comment:14 Changed 7 years ago by neverpanic (Clemens Lang)

I've been working on getting this working, ideally even without modifications to the master.cfg of the buildbot, but I've hit some problems. [f6e46815a5033358212ce383956fd567eb51e53c/mpbb] changed the variant handling code in mpbb in a way that will cause failures when requesting specific variants to be installed. See the commit on GitHub for a description of the problem. We would have to revert this commit for this effort to proceed, but I'd like to wait for feedback from Joshua before doing that.

comment:15 Changed 7 years ago by neverpanic (Clemens Lang)

comment:16 Changed 7 years ago by jmroot (Joshua Root)

I don't doubt the current code needs changes to work right when specifying arbitrary variants. Simply reverting that commit is not right either, though. The distinction between the canonical_active_variants and the variants that are being requested needs to be maintained.

comment:17 Changed 7 years ago by neverpanic (Clemens Lang)

You're right. I thought about this yesterday and came to the conclusion that this is really a bug in base, since base propagates these variants down the dependency tree and attempts to install a combination that will not work.

Either the glib2 Portfile should ignore the user's request to change the variant and forcefully set a variant, or base should not request this invalid combination.

comment:18 Changed 7 years ago by neverpanic (Clemens Lang)

In 45c59fe0a66e382927fbf6fe629d37898eb42b6b/macports-ports:

glib2: Default to +quartz when -x11 is given

When building pango -x11 in a clean prefix, glib2 will fail to install,
because the -x11 variant is being passed down to glib2, but glib2
requires either +quartz or +x11, and +quartz was not enabled by default
when x11 was disabled.

This caused problems when implementing variant support for the buildbot
in https://github.com/macports/mpbb/pull/5 and initially caused me to
think that I would have to revert

f6e46815a5033358212ce383956fd567eb51e53c/mpbb

which was added due to https://github.com/macports/mpbb/pull/4 and
https://lists.macports.org/pipermail/macports-dev/2017-June/035978.html.

This solution should instead work without the revert and still allow the
buildbot to build both wine and +quartz-x11 ports.

See: #52742

comment:19 Changed 7 years ago by neverpanic (Clemens Lang)

Owner: changed from admin@… to neverpanic
Status: newassigned

comment:20 Changed 7 years ago by ryandesign (Ryan Carsten Schmidt)

In 32fd667ad3d4cb8fe9be7bf2dc5058cc73d8e530/macports-ports:

glib2-devel: Default to +quartz when -x11 is given

When building pango -x11 in a clean prefix, glib2 will fail to install,
because the -x11 variant is being passed down to glib2, but glib2
requires either +quartz or +x11, and +quartz was not enabled by default
when x11 was disabled.

This caused problems when implementing variant support for the buildbot
in https://github.com/macports/mpbb/pull/5 and initially caused me to
think that I would have to revert

f6e46815a5033358212ce383956fd567eb51e53c/mpbb

which was added due to https://github.com/macports/mpbb/pull/4 and
https://lists.macports.org/pipermail/macports-dev/2017-June/035978.html.

This solution should instead work without the revert and still allow the
buildbot to build both wine and +quartz-x11 ports.

See: #52742

Note: See TracTickets for help on using tickets.