Mirror distfiles immediately after a commit

[ryandesign (Ryan Carsten Schmidt)]

Thought we had a ticket for this already but can't find it.

When GitHub notifies our buildbot 0.8 setup of a commit, it schedules a portwatcher build for each OS version. One of the steps of portwatcher is to trigger a mirror job.

This arrangement causes 2 problems:

1. A mirror job is scheduled for each OS version. We only need it done once because the mirror job already loops over each OS version to download any files specific to that OS version. Scheduling one mirror job for each OS version wastes time at best, and at worst can cause extraordinary amounts of unnecessary network traffic for the buildbot and the servers it contacts; see #57343.
2. Files are not mirrored until right before they're built. This is a problem if all of the builders for 10.9+ are busy. For example, right now, all builders except one have been busy for days building all the p5.30 ports. Commits for other port updates happened after that, but they are waiting, and so their files have not been mirrored. Meanwhile, users of 10.6-10.8 will learn of those updates when they port sync, but they might not be able to upgrade their ports if upstream uses restrictive SSL. Our 10.6-i386 builder is behind the others and has not yet reached the p5.30 ports commit, so it is still working through older updates. But because the buildbot updates the ports tree to current before each build, it already knows about the sqlite3 update, and tries to build it as a dependency of other ports, but fails to fetch it, so I have had to take that builder offline.

We should do things the other way around. On GitHub notification of a new commit, we want to schedule one mirror job. When that mirror job is done, it should trigger (and not wait for) a portwatcher build for each OS.

Portwatcher needs to be enhanced to be able to distinguish between being triggered from a GitHub notification and being forced from the web interface. When forced from the web interface, portwatcher should trigger a mirror job as it does now. When notified from GitHub, it should not.

I know work has already been done on a buildbot 2 setup. Does this work already address this issue?

[jmroot (Joshua Root)]

This was touched on in wiki:Meetings/MacPortsMeeting2018/BuildbotRestructuring as well.

[rajdeepbharati (Rajdeep Bharati)]

I know work has already been done on a buildbot 2 setup. Does this work already address this issue?

The work doesn't already address this issue, but this would be pretty easy to implement IMHO.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

2. Files are not mirrored until right before they're built. This is a problem if all of the builders for 10.9+ are busy. For example, right now, all builders except one have been busy for days building all the p5.30 ports. Commits for other port updates happened after that, but they are waiting, and so their files have not been mirrored. Meanwhile, users of 10.6-10.8 will learn of those updates when they port sync, but they might not be able to upgrade their ports if upstream uses restrictive SSL. Our 10.6-i386 builder is behind the others and has not yet reached the p5.30 ports commit, so it is still working through older updates. But because the buildbot updates the ports tree to current before each build, it already knows about the sqlite3 update, and tries to build it as a dependency of other ports, but fails to fetch it, so I have had to take that builder offline.

In fact it's worse than this: the files might never get mirrored at all.

Consider: During the days that the p5.30 ports were being built, sqlite3 was updated. After the p5.30 builds finished on the 10.9 builder, a build of another port started, which had sqlite3 somewhere in its dependency chain, so sqlite3 got built and deployed—but not mirrored, because it wasn't in the list of portfiles modified in that commit. Later, when the builder got around to ​the commit that actually updated sqlite3, it saw that sqlite3 had already been deployed and ​excluded it from the list. The distfile would not have been mirrored at all, were it not for the fact that the sqlite3 port happens to have other subports that still needed to be built.

If we mirrored distfiles right away, before scheduling builds, we would avoid that problem as well.

[mojca (Mojca Miklavec)]

I have the following building scheme in mind:

+--------------+
|-  WATCHER   -|
+--------------+
|  (commit)    |
+--------------+
| 1-mprsyncup  |
+--------------+
| 2-mirror     |
+--------------+         +----------------+      +----------------+     +-----------------+
| 3-scheduler  |  ---->  |- 10.6 builder -|      |- 10.7 builder -| ... |- 10.15 builder -|
| (no waiting) |         +----------------+      +----------------+     +-----------------+
+--------------+         | svn up         |      | svn up         |     | svn up          |
                         +----------------+      +----------------+     +-----------------+
                         | clean          |      | clean          |     | clean           |
                         +----------------+      +----------------+     +-----------------+
                         | selfupdate     |      | selfupdate     |     | selfupdate      |
                         +----------------+      +----------------+     +-----------------+
                         | port list      |      | port list      |     | port list       |
                         +----------------+      +----------------+     +-----------------+
                             +----------------+      +----------------+     +-----------------+
                             |- dep1_10.6    -|      |- dep1_10.7    -|     |- dep1_10.15    -|
                             +----------------+      +----------------+     +-----------------+
                             | dep1 install   |      | dep1 install   |     | dep1 install    |
                             +----------------+      +----------------+     +-----------------+
                             | dep1 archive   |      | dep1 archive   |     | dep1 archive    |
                             +----------------+      +----------------+     +-----------------+
                             | dep1 upload    |      | dep1 upload    |     | dep1 upload     |
                             +----------------+      +----------------+     +-----------------+
                             | dep1 deploy    |      | dep1 deploy    |     | dep1 deploy     |
                             +----------------+      +----------------+     +-----------------+
                             | clean          |      | clean          |     | clean           |
                             +----------------+      +----------------+     +-----------------+
                             +----------------+      +----------------+     +-----------------+
                             |- dep2_10.6    -|      |- dep2_10.7    -|     |- dep2_10.15    -|
                             +----------------+      +----------------+     +-----------------+
                             | dep2 install   |      | dep2 install   |     | port3 install   |
                             +----------------+      +----------------+     +-----------------+
                             | dep2 archive   |      | dep2 archive   |     | port3 archive   |
                             +----------------+      +----------------+     +-----------------+
                             | dep2 upload    |      | dep2 upload    |     | port3 upload    |
                             +----------------+      +----------------+     +-----------------+
                             | dep2 deploy    |      | dep2 deploy    |     | port3 deploy    |
                             +----------------+      +----------------+     +-----------------+
                             | clean          |      | clean          |     | clean           |
                             +----------------+      +----------------+     +-----------------+
                             +----------------+      +----------------+
                             |- port1_10.6   -|      |- port1_10.7   -|
                             +----------------+      +----------------+
                             | port1 install  |      | port1 install  |
                             +----------------+      +----------------+
                             | port1 archive  |      | port1 archive  |
                             +----------------+      +----------------+
                             | port1 upload   |      | port1 upload   |
                             +----------------+      +----------------+
                             | port1 deploy   |      | port1 deploy   |
                             +----------------+      +----------------+
                             | clean          |      | clean          |
                             +----------------+      +----------------+

with the "displaced" builds being done by virtual builders.

I would say that it makes sense to try the effort of deploying the version with buildbot 2 as soon as possible rather than spend significant amounts of time fixing the 0.8 setup.