Redesign R ports/portgroup so that a revbump is not needed after major R updates

[ryandesign (Ryan Carsten Schmidt)]

Revbumping all 4,854 R ports after a major R update, as was done in [6fde9135d5a6eecb1f719535588f4812bc01431b/macports-ports] and [2329655ef76aaf38dcd204702ba981a6169b77b3/macports-ports], completely incapacitates the buildbot. It's not just that building that many ports will take weeks, if everything works perfectly. It's that buildbot (version 0.8, at least) is not designed to handle that many pending builds. I don't know if there's a missing SQLite index or some other problem but last year when this happened, with thousands of pending builds on each builder, it took literally hours for the buildmaster to decide which pending build to start next each time, during which time all the builders sat idle. A quick back-of-the-envelope calculation suggests that, left unattended, that would take a decade to resolve itself. It took me hours of manual work to cancel all the jobs to make the buildbot functional again. Oh, and last year it was only 2,047 ports.

This time, fortunately most of the workers were busy on other builds and I was able to remove the commit from the queue before it was processed, and the ones that were already processing it had only gotten to the mirroring step (which by itself takes hours—not to mirror the files, just to verify that they're already mirrored) and were easy to cancel before the individual port builds got scheduled. I then loaded the list of changed ports into a script that I use to mass build ports in small batches (a script usually used to build all ports when a new macOS version is released) like I did last time. These batches will probably take a month to complete, since a new batch isn't scheduled unless the builder is almost idle, but at least the buildbot will be functional and building things during that time. The "reason" field in each batch will show how many ports remain to be scheduled on that builder.

Please redesign the R ports/portgroup so that revbumping all of them at once is not needed again (with the exception of one final mass revbump to remove the R version number from the install path).

If the version number in the install path is not the only problem and such a redesign is not possible, then such mass revbumps need to be coordinated with the buildbot administrator which would be me. I would probably need to be the one to merge the pull request, ensuring that GitHub webhooks are turned off beforehand and turned back on afterward, and then I would need to set up and run the batch script.

[jmroot (Joshua Root)]

I'm not clear on why we even have 4,854 R ports. Is anyone using most of them? Can't they be installed via CRAN? Python is one of the most popular languages, and we have 2,278 entries in the python category, since we tend not to package modules that can easily be installed with pip and aren't needed by ports.

[barracuda156]

Replying to ryandesign:

It is not the PG, it is R itself which sets the path. PG simply mimics that to have a correct install location. But I guess we can patch the R so skip minor version component (or replacing it by x), and then the only time revbump will be needed is a change in major version (which is justified and happens once in a blue moon).

Since rebuilds are paused, let’s do this today? I can handle R and PG, and then you handle batches of updates. And we won’t face this issue until R 5.x.

[barracuda156]

Replying to jmroot:

I'm not clear on why we even have 4,854 R ports. Is anyone using most of them? Can't they be installed via CRAN? Python is one of the most popular languages, and we have 2,278 entries in the python category, since we tend not to package modules that can easily be installed with pip and aren't needed by ports.

As you can see, many other downstream distributions have their ports for R stuff instead of just relying on CRAN. There are a few issues with using CRAN, some of which are easy to solve, some perhaps non-trivial and some are completely breaking.

What we have is working better than CRAN.

Having said that, I agree that given what Ryan said, we should avoid revbumps on minor version upgrades. So that will be done, I will deal with this today.

[barracuda156]

Replying to ryandesign:

So this is controlled by FW_VERSION in configure script of R. I will sort this out so that we do not face this issue again.

16. S. The title should actually refer to R port, since the rest it just following how R is installed. This is not a problem of PG or packages.

[barracuda156]

​https://github.com/macports/macports-ports/pull/23732