github: CI jobs: port dependency installation incredibly slow, severely impacting job runtime

[mascguy (Christopher Nielsen)]

Per discussion on the Dev mailing list during April 2021, port dependency installation within our GitHub CI jobs is incredibly slow. Enough so that it severely impacts the job runtime, and in the worst cases, it causes the jobs to be killed due to timeout. (The latter is presently six hours.)

This also has a further cascade effect, causing CI jobs to be queued up. And increasing the wait time for PRs to be validated.

This needs to be improved, if at all possible.

Also see issue:62621, which covers dependency installation time for our buildbots.

[mascguy (Christopher Nielsen)]

One of the examples cited was for port nomacs:

The first dependency started installing at 3:33pm:
2021-04-24T15:33:40.0051560Z Installing dependency (1 of 265) 'bzip2' with variants '' (requesting '') ... [OK]

While the last dependency finished installing at 5:27pm, two hours later:
2021-04-24T17:22:49.4607320Z Installing dependency (265 of 265) 'qt5-qttools' with variants '' (requesting '') ... [OK]
2021-04-24T17:27:26.6862050Z ##[endgroup]

[mascguy (Christopher Nielsen)]

Mailing list discussion thread:

​https://lists.macports.org/pipermail/macports-dev/2021-April/043259.html

[ryandesign (Ryan Carsten Schmidt)]

Discussion starting at comment:ticket:62621:4 covers this.

[mascguy (Christopher Nielsen)]

Another example from today, 2021-05-17, for port py-apache-beam; ​PR 11044 - py-apache-beam: Update to version 2.29.0:

2021-05-17T16:42:49.3114950Z Installing 342 dependencies of py39-apache-beam:
[...]
2021-05-17T19:17:38.8274350Z Installing dependency (342 of 342) 'py39-pymongo' with variants '' (requesting '') ... [OK]
2021-05-17T19:24:21.8912600Z ##[endgroup]

Dependency installation time was 2 hours, 42 minutes, for the first subport build.

Full log attached; filename: github-ci-py-apache-beam-20210517.txt.gz.

[mascguy (Christopher Nielsen)]

Large dependencies - specifically, ports consisting of many thousands of files, like the various texlive-related components - would also benefit from speedup of the file registration process (issue:56793).

[mascguy (Christopher Nielsen)]

Update: Josh's fix for issue:56793 will improve dependency installation times in some situations. And for the most extreme cases - ports that depend on huge components like texlive-fonts-extra - that fix is a game-changer! (Both for buildbot/CI jobs, as well as the user experience in-general.)

Net-Net: For ports like auto-multiple-choice, which fit into the latter category, we'll see overall dependency installation time reduced dramatically. Great stuff!