buildbot: allow building 'pseudoports'

[mojca (Mojca Miklavec)]

I would find it useful if we would allow building a subset of ports with simple expressions, so that we could rebuild all ports (either in one go or incrementally) every now and then.

In particular it would be helpful if we could build category:shells or all or similar things.

We are currently using the following expression to list the subports:

port info --index --line --name wxWidgets-3.0 subportof:wxWidgets-3.0

The following works:

port info --index --line --name 'category:shells'

but

port info --index --line --name 'category:shells' subportof:'category:shells'
port info --index --line --name all subportof:all

doesn't.

What would be needed to get this working?

After inspecting the source code I see that a list of ports is generated with

for p in "$@"; do
    print-subports "$p" && success=1
done

and print-subports does

ports=$("${option_prefix}/bin/port" -q info --index --line --name "${portname}" "subportof:${portname}" 2>/dev/null) || return

which fails if the argument of subportof: is not a proper port name. One possible solution would be to run port info first, for example

> port info --index --line --name maintainer:mojca and category:math
aquaterm
gnuplot
libcerf

or

> port info --index --line --name wxWidgets-2.8 wxWidgets-3.0
wxWidgets-2.8
wxWidgets-3.0

and then iterate over that result rather than on the list of arguments (three in the example maintainer:mojca and category:math) to print all subports. The only problem is that category:foo would already list all subports, so additional measures would be needed to make sure that we don't end up with duplicates of build jobs.

At the same time we should be a bit careful about code injections to prevent logged in users from executing arbitrary shell code on the build slaves.

Any thoughts?

[larryv (Lawrence Velázquez)]

I don’t think this is a technical problem at all; it seems simple to rework mpbb to handle this safely. The actual question is whether we want to allow people to do this. Then: Why stop at pseudoports? Why not allow arbitrary port(1) expressions, including Boolean operators and grouping parentheses?

[mojca (Mojca Miklavec)]

We are not discussing about allowing random users to do this, but allowing a limited set of trusted developers to do so.

We are not discussing adding anything that's not already possible: these users can already copy paste a super long list of ports to the force build field.

What I would like is a simple way to do things that are otherwise very tedious to do.

We've been holding off the commit with $Id field for some time also because we would like to misuse that to rebuild all ports and were afraid to do so without success cache is place. I would prefer if we had an official easy way to launch a rebuild of some/all ports without tedious copy-pasting of tens of thousands of ports.

[larryv (Lawrence Velázquez)]

I would argue that force-building large numbers of ports should be a rare and significant event that ought not be easy to do at all.

[mojca (Mojca Miklavec)]

How exactly would you handle the situation after a new build slave gets added?

I'm not saying that we should do that on a weekly basis. But I claim that current situation is worse. Some people are in favour of letting the builds go through after modifying all ports. At the moment we have zero experience of the consequences of such action. It would be much much much safer if we could try building a subset of ports (but still a large enough sample) to even get a rough idea of problems that we might want to address before trying to let such a huge build go through.

Some people also say that we should let the build go through because that's a rare occasion that lets us fill in the missing builds, also because it's currently not easy to do so. In my opinion it would be better if we could start partial builds ten times rather than doing them all at once.

A random committer may already just touch all ports (for example add a blank line at the end; or replace the formatting of maintainer or anything else, ...). So it's already "easy enough".

[mojca (Mojca Miklavec)]

And yes, I wouldn't mind support for arbitrary expressions. It doesn't seem to be any more difficult to support (as shown in the above example).

The only thing that we need to make sure is not to make it too easy to accidentally end up with too many ports being built.

[larryv (Lawrence Velázquez)]

Replying to mojca:

How exactly would you handle the situation after a new build slave gets added?

I don’t know. What have we been doing so far?

[larryv (Lawrence Velázquez)]

Replying to mojca:

A random committer may already just touch all ports (for example add a blank line at the end; or replace the formatting of maintainer or anything else, ...).

Just because it is possible in some fashion does not mean we endorse it. Allowing pseudoports in the Buildbot interface would be an explicit endorsement of large builds. It says “we have designed our UI to permit this and make it easy”.

[mojca (Mojca Miklavec)]

Replying to larryv:

Replying to mojca:

How exactly would you handle the situation after a new build slave gets added?

I don’t know. What have we been doing so far?

Ryan often built all ports (maybe in batches?). I'm not sure how exactly he did it. Maybe he came up with a list of all ports (or a list of directories) and split it in parts. Or at least I believe he did that on the old buildbot where this was less problematic. I don't think we ever did it on the new infrastructure so far, I'm only aware of ​this build of python modules plus some hand-picked list of ports to make life on 10.5, 10.7 and 10.12 easier.

Also note that manual interventions are pretty rare and not many developers really care about forcing builds, I wouldn't worry about problems here.

[ryandesign (Ryan Carsten Schmidt)]

The old macOS forge buildbot allowed specifying "all" as a special-case port name. That may have worked fine when the old buildbot was first set up, but in the years since then our ports collection grew such that building all ports at once became problematic as the amount of output produced was more than buildbot was designed to handle, resulting in weird errors and requiring us in some cases to re-issue the build a second time, wasting time.

On the new buildbot, we haven't ever built all ports, even though we've set up four completely new workers in that time and resurrected one worker that was offline for a year on macOS forge, so there are probably many ports that these five workers could build but have never been asked to. The new buildbot uploads not only the port being built, but also any dependencies that got built, so in that way we are able to quickly build a number of useful binaries just by letting developers' commits trigger builds of those ports that are actually being used. Still, it would be nice to be able to trigger builds of all ports, to ensure we have actually tried to build all the ports we could. To reduce the amount of time that the builder is out of commission for other work, it would be nice to be able to split it into groups (e.g. by category or name) without having to do that work on the command line and paste a long list of ports into the portname field.

[mojca (Mojca Miklavec)]

Adding a new builder was precisely the main reason why I opened this ticket. Solving this should be pretty straightforward with a few lines of shell scripting (or perhaps a tiny addition to the port command to allow for easier listing of subports), only a few lines in mpbb need a fix.

I don't know how to sanitize input though.

[ryandesign (Ryan Carsten Schmidt)]

Which pseudo-portnames would be useful to support? On the old system we only supported "all" but maybe it would be better to use "uninstalled".

[mojca (Mojca Miklavec)]

My hope was to support any expression that's supported by MacPorts. Like in the example above, it would be really nice if I was able to write something like "maintainer:mojca and category:math" in the portname field. Then we could do arbitrary expressions and build ports by categories or by their first letter or whatever else.

If uninstalled means what I think it does, then we could theoretically easily build all ports that depend on some-importart-port-that-has-been-broken-for-a-week which have not been built yet. (I don't know if we can currently construct the list of dependent ports; maybe not yet.)

If implemented properly, we'll get all the flexibility nearly for free.