Many binary packages are missing for Monterey x86-64

[FranklinYu (Franklin Yu)]

Example: ​squirrel-ime, ​librime-devel, ​fortune.

This seems to affect all Rust ports, including ​ripgrep, ​bat, ​exa.

[jmroot (Joshua Root)]

Yes, that's entirely expected since the builder hasn't yet gotten through the commit backlog. Once it does, builds can be started for ports that have not yet been built. Until that is done, anything that hasn't been updated since this builder was added will not be built.

[ryandesign (Ryan Carsten Schmidt)]

I have a script running that schedules small batches of ports that have not yet been attempted, when the builder is not otherwise occupied with incoming commits.

[FranklinYu (Franklin Yu)]

Replying to jmroot:

Yes, that's entirely expected since the builder hasn't yet gotten through the commit backlog. Once it does, builds can be started for ports that have not yet been built. Until that is done, anything that hasn't been updated since this builder was added will not be built.

Thank you. Is there anywhere online that we can see the progress? For example, which commit is the builder currently building?

[jmroot (Joshua Root)]

​https://guide.macports.org/chunked/development.buildbot.html

[FranklinYu (Franklin Yu)]

Replying to jmroot:

​https://guide.macports.org/chunked/development.buildbot.html

Would you mind being more specific about which dashboard to look at? I see that ​https://build.macports.org/builders says ports-12_x86_64-builder has “28 pending”; is this the best indicator, for the backlogs you mentioned in comment:1?

[jmroot (Joshua Root)]

Sure. Ports-*-watcher are where incoming commits are processed, and they determine what needs to be built based on each commit and queue builds on the corresponding builder. Those are all more or less caught up now, and Ryan's script continues to operate.

[FranklinYu (Franklin Yu)]

On ​https://build.macports.org/builders/ports-12_x86_64-watcher I saw

A build was forced by 'ryandesign <ryandesign@localhost>': force build

Is this what Ryan mentioned in comment:2? Is this what I should be looking at, if I want to know the backlog?

[ryandesign (Ryan Carsten Schmidt)]

Replying to FranklinYu:

On ​https://build.macports.org/builders/ports-12_x86_64-watcher I saw

A build was forced by 'ryandesign <ryandesign@localhost>': force build

Is this what Ryan mentioned in comment:2?

Yes.

Is this what I should be looking at, if I want to know the backlog?

No. The backlog Josh referred to earlier was due simply to commits made by MacPorts contributors that the 12_x86_64 builder had not yet gotten around to processing. Josh made improvements to mpbb that reduced the amount of time it spends on some build tasks, allowing that backlog, which had existed since macOS Monterey was introduced, finally to clear and for the builder to better keep up with the volume of commits that occur daily. Now that the backlog has cleared, the builder got close to being idle and my script was able to begin scheduling small batches of builds from the list of ports that had not yet been attempted due to commits. That list currently has about 5500 non-perl ports left to attempt, plus about 1500 perl 5.34 modules (perl 5.34 and all its modules were just added but the commit that added them was removed from the normal scheduler because processing that one commit would have taken days or weeks during which no other builds could take place), plus about 4500 non-5.34 perl modules, most of which have already been built but which will be run through the scheduler again in case any can now be built or distributed that could not be built or disributed last time they were tried (since the last time they were tried, openssl was updated to version 3, which has a better license which allows more ports that depend on openssl to be distributable).

[FranklinYu (Franklin Yu)]

I see, but then how can I tell whether your script has cleared its list of pending ports (I mean the 5.5k + 1.5k + 4.5k mentioned in comment:8)? Since you called it a “script”, I assume we didn’t built any web interface for it. Looking at the list of builders, the watcher queue-length doesn’t seem like a good indicator either: the 10.13 (High Sierra) watcher has a queue of comparable length.

[ryandesign (Ryan Carsten Schmidt)]

There is no way for anyone other than me to see how many port names are remaining in the list. Currently, the list has about 10,500 port names remaining.

[ryandesign (Ryan Carsten Schmidt)]

My forced buildsets now set the "reason" field to show how many items are remaining after that set.

​https://build.macports.org/builders/ports-12_x86_64-watcher

[FranklinYu (Franklin Yu)]

I checked several ports I use, and they all have binary packages now. Thank you very much for the script!