distfile mirrors not automatically updating

[GregoryEAllen (Greg Allen)]

Ryan Schmidt wrote:

We have not yet brought automatic distfile mirroring online on the new server after moving away from macOS forge.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #53423.

To explain: many sites now only offer downloads via https, and the versions of curl and openssl in Leopard and Tiger are too old to be able to communicate with modern https sites, so fetching the files from the original site fails. This wouldn't be a problem if we had our http distfile mirroring up and running, so we need to get it back online again.

[noloader (Jeffrey Walton)]

Replying to ryandesign:

... To explain: many sites now only offer downloads via https, and the versions of curl and openssl in Leopard and Tiger are too old to be able to communicate with modern https sites, so fetching the files from the original site fails. This wouldn't be a problem if we had our http distfile mirroring up and running, so we need to get it back online again.

If it helps, I keep a handful of utilities up-to-date in /usr/local/bin as a fallback. They include cURL, OpenSSL, SCP, SSH and Wget. They all use static linking to avoid dll hell and versioning problems.

Maybe MacPorts could check for an updated (more recent?) version in /usr/local. Or, allow me to specify an override in /opt/local/etc/macports/macports.conf:

curl_bin     = /usr/local/bin/curl
openssl_bin  = /usr/local/bin/openssl
wget_bin     = /usr/local/bin/wget

[kencu (Ken)]

be careful. /usr/local/include and /usr/local/lib are often on the search paths for macports compilers. Strange and hard-to-debug compiler and build errors can occur if the wrong include or lib files are found by macports compilers during the build. Builds can fail with weird errors that make no sense.

You might be best to put your own files somewhere like ~/bin where nothing will go looking for them. Up to you.

[raimue (Rainer Müller)]

Replying to noloader:

If it helps, I keep a handful of utilities up-to-date in /usr/local/bin as a fallback. They include cURL, OpenSSL, SCP, SSH and Wget. They all use static linking to avoid dll hell and versioning problems.

This ticket is about getting infrastructure ready to do the automatic mirroring on the MacPorts buildbot instance. For the problems with HTTPS and an old libcurl, please see #51516.

[GregoryEAllen (Greg Allen)]

I use distfiles because I work on networks that can't be directly connected to the internet, which is common in government research labs. Mirrors are fairly easy to set up and manage, but direct access is disallowed. I'm currently stuck with manually copying packages via "sneakernet".

[kencu (Ken)]

I'm looking to understand why distfile mirroring doesn't work at present. Was this feature implemented as part of MacPorts by using the portmirror:mirror_main function in /opt/local/libexec/macports/lib/port1.0/portmirror.tcl? If so, I don't immediately see why it is not working...

Or was there perhaps a specific buildbot with a current curl that mirrored it's distfiles to some main repository source mirror using a cron/rsync script outside of macports?

[jmroot (Joshua Root)]

Running 'port mirror' works fine, it just isn't set up to happen automatically on distfiles.macports.org.

[kencu (Ken)]

OK, so that part is working.

I take it there used to exist a cron job running a script on a similarly-named server that ran the whole list of portnames through port mirror portname every once in a while?

And we are missing either that script (it must still exist somewhere, I suspect), and the cron job entry? This is sounding encouraging...

[kencu (Ken)]

Could it be that this cronjob is all we need on the distfiles.macports.org server?

30 * * * * /opt/local/bin/port -p mirror all 2>&1 >> /var/log/macports-mirror.log

[jmroot (Joshua Root)]

No, that's not sufficient. Mirroring all ports takes too long, we aren't allowed to mirror some ports, and we have to make sure that other jobs don't update the copy of macports being used while mirroring is in progress. We also ideally want to mirror extra distfiles that may be added by variants.

The macports-infrastructure repo has the work-in-progress code.

[kencu (Ken)]

Got it -- it would appear this is the script that needs to be hit by the cron job. macports-infrastructure/jobs/mirror_macports.sh​.

[ryandesign (Ryan Carsten Schmidt)]

The old macOS forge infrastructure mirrored distfiles in a Subversion post-commit hook, prior to sending the commit to the buildbot. Thus, the buildbot workers could download their distfiles from our mirror. This is what we want to achieve again, except now we're using GitHub, so we don't have a local post-commit hook script, and instead we get notifications from GitHub which trigger the buildbot. So we want to create a buildbot task that mirrors the files prior to scheduling the port-building task, or we want to have the port-building tasks depend on the mirroring task. Buildbot tasks occur in what the Buildbot project calls "builders", and that builder needs to be running on the server hosting the distfiles. We currently don't have a Buildbot builder running there, so I need to set one up, and decide what user to run that as. That builder also needs a copy of MacPorts and the ports tree, and for that copy of MacPorts and the ports tree to be kept up to date; I don't think we've yet solved how that happens.

[kencu (Ken)]

It appears that most of this logic is in ​buildbot/master.cfg. There are entries in there that appear to reference the mirror stage.

if 'mirror' in config['deploy']:
    jobs_mirror_factory = util.BuildFactory()
    jobs_mirror_factory.addStep(steps.ShellCommand(
        command=util.WithProperties('port mirror %(portname)s'),
        name='mirror',
        description=['mirroring'],
        descriptionDone=['done']))

[ryandesign (Ryan Carsten Schmidt)]

Automatic distfile mirroring is online on our buildbot now. It happens at commit time, so as ports are updated from now on, their files (and those of their dependencies) will be mirrored.

Existing ports that are not modified and that are not dependencies of modified ports will not be automatically mirrored. To partially address this, I will do some batch modifications of ports that I've been putting off until mirroring was fixed. I will also attempt to add the ability for a developer to force a port's files to be mirrored by the buildbot (without forcing the buildbot to build the port). I'll then investigate the feasibility of initiating a one-time mirroring of all ports.

[kencu (Ken)]

Thanks very much to all for the effort involved in this. It is much appreciated.