Set up a private server for archives that aren't distributable to speed up Travis CI builds

[yan12125 (Chih-Hsuan Yen)]

Motivation

I just submitted a new port libfilezilla. Builds on Travis CI timed out after 48 minutes [1] while on buildbot it takes about 8 minutes only [2]. I believe what @raimue said [3] - time to build non-redistributable dependencies - is the key difference. I hope we can have a caching mechanism so that newcomers can know if their patches are OK or not quickly, and maintainers can spend less time on testing patches manually.

Proposal

Travis CI provides a simple caching mechannism [4]. If built packages under $PREFIX/var/macports/software are cached, there's no need to rebuild them every time.

Cached data are only accessible to Travis CI builders. There are no URLs to cached files in build logs. I guess it's not a "redistribution" as other people can't access them. Thus there are no licensing issues.

[1] ​https://travis-ci.org/macports/macports-ports/builds/273323633

[2] ​https://build.macports.org/builders/ports-10.12_x86_64-builder/builds/38131

[3] ​https://github.com/macports/macports-ports/pull/756#issuecomment-328345604

[4] ​https://docs.travis-ci.com/user/caching/

[l2dy (Zero King)]

1. $PREFIX is not writable by the user (​https://docs.travis-ci.com/user/caching/#caches-and-read-permissions). But this is easy to workaround by moving files over and fixing permissions.
2. In order to cache dependencies across PRs, we have to enable builds on the master branch. To avoid timeouts, we can only build ports changed in the last commit pushed. (When many unrelated commits are pushed at once, only one Travis build will be triggered.)
3. We should not cache distributable packages and purge outdated cached packages to avoid making the cache too big. Extracting the cache (and maybe generating it) costs precious build time.

After the first pull request build is run, it creates a new pull request cache.

4. We have to remove cache of ports to be tested in PRs.

[yan12125 (Chih-Hsuan Yen)]

Thanks for comments. Here are some numbers for the caching speed. In a recent build of my project, it takes 19.37s to download and 11.05s to extract a 3554.06MB cache. Regenerating the cache takes 65.66s. Indeed huge caches bring problems.

[l2dy (Zero King)]

Travis CI cache is not suitable for this (hard to implement, too many constraints).

Hosting these binaries on a private (limiting access to Travis CI VMs only) server technically works but I'm not a lawyer so...

[mojca (Mojca Miklavec)]

That's not a question for lawyers. We theoretically have the same issue with buildbot in case a builder runs out of space. It would probably make sense to store packages on a private server both for Travis and building.

[ryandesign (Ryan Carsten Schmidt)]

Does this look like a reasonable list of steps to accomplish this?

• On the buildbot server, set up a new directory outside of the rsync directory to store nondistributable archives.
• Change mpbb so that it uploads nondistributable archives to there. (We don't need to worry about cleaning up old nondistributable archives; the existing cleanup script can probably be used for that.)
• At NameCheap, make 2 new hostnames on macports.org for this: one for the origin (on the buildbot server) and one for the CDN.
• On the buildbot server, configure nginx to serve the new directory via that new origin hostname (and also a local-network-only hostname for use by the buildbot), and to allow access only from MaxCDN IP addresses (and localhost).
• At MaxCDN, add a new zone for this with the new CDN hostname and configure the zone to allow access only from Travis IPs.
• Modify Travis config to also find packages at the new CDN hostname by adding an entry to archive_sites.conf.
• Modify buildbot config to also find packages at the new local-network-only hostname by adding an entry to archive_sites.conf.
• Maybe the preceding 2 happen in mpbb; if so, do we have a way in mpbb to differentiate whether we're running on Travis or buildbot?

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

• On the buildbot server, set up a new directory outside of the rsync directory to store nondistributable archives.

Done. I've used a separate script to manually populate this directory with all current archives from all builders. After buildbot is modified to upload nondistributable archives there, I can run the separate script once more to sync everything up, then run another script to remove distributable archives from this directory, leaving only the nondistributable ones.

• Change mpbb so that it uploads nondistributable archives to there. (We don't need to worry about cleaning up old nondistributable archives; the existing cleanup script can probably be used for that.)

​https://github.com/macports/mpbb/pull/9

Buildbot configuration changes are also needed to supply the new arguments to mpbb.

​https://github.com/macports/macports-infrastructure/pull/8

• At NameCheap, make 2 new hostnames on macports.org for this: one for the origin (on the buildbot server) and one for the CDN.

Done.

Still to do: automating the renewal of the Let's Encrypt SSL certificates.

• On the buildbot server, configure nginx to serve the new directory via that new origin hostname (and also a local-network-only hostname for use by the buildbot), and to allow access only from MaxCDN IP addresses (and localhost).

Done.

• At MaxCDN, add a new zone for this with the new CDN hostname and configure the zone to allow access only from Travis IPs.

Done.

• Modify Travis config to also find packages at the new CDN hostname by adding an entry to archive_sites.conf.

Still to do. Another possibility is setting archive_site_local in macports.conf.

• Modify buildbot config to also find packages at the new local-network-only hostname by adding an entry to archive_sites.conf.

The problem is that mpbb has its own separate copy of macports.conf, which references its own separate copy of sources.conf. However, I don't see any way to reference a separate copy of archive_sites.conf. Using archive_site_local in macports.conf is an alternative; this is included in ​https://github.com/macports/mpbb/pull/9. However, it means that the standard set of archive sites is still used; I would like to find a way to disable that.

• Maybe the preceding 2 happen in mpbb; if so, do we have a way in mpbb to differentiate whether we're running on Travis or buildbot?

mpbb sets up macports.conf in mpbb-checkout. Travis doesn't use mpbb-checkout, so at present these changes will only affect buildbot. I've added arguments so that the archive site URLs can be specified when calling mpbb-checkout, so that Travis and Buildbot could request different URLs.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

• Modify Travis config to also find packages at the new CDN hostname by adding an entry to archive_sites.conf.

Still to do. Another possibility is setting archive_site_local in macports.conf.

​https://github.com/macports/macports-ports/pull/3099

[ryandesign (Ryan Carsten Schmidt)]

In b34796f273fe84e4fd6388cbaf7ac7e7bed7b8d4/macports-ports (master):

Also allow Travis to download private archives

See: #54800

[ryandesign (Ryan Carsten Schmidt)]

In 4dad85ab6d42f6feed7457a62fe45ed648a46660/macports-ports (master):

Prefer public archives over private archives

Right now, while we're setting things up, the private server also has
the public archives, but we don't want to waste time adding those to
the CDN cache when they're already in the public server's CDN cache.

See: #54800

[ryandesign (Ryan Carsten Schmidt)]

In e74b76fe006d4227fdc618112e51125ec82ae1da/macports-ports (master):

Specify archive_site_local correctly, with :tbz2

See: ​https://github.com/macports/macports-ports/pull/3099
See: #54800

[ryandesign (Ryan Carsten Schmidt)]

In b00a34817be1ff2a77f5acc63dc8adafb8993179/macports-ports (master):

Fix archive_site_local base bug for Travis

See: #57717
See: #54800
See: ​https://github.com/macports/macports-ports/pull/3099

[ryandesign (Ryan Carsten Schmidt)]

In 531720bba7af24204b507f26bdefbc60e58556c0/macports-ports (master):

Travis: Prefer public archives over private ones

I had tried to do this with preferred_hosts before, but it appears to
have no effect on archive_site_local.

See: #57720

Instead list both the public and the private URLs in archive_site_local.
They appear to be tried in the order listed.

But then we need to fix the base bug with multiple URLs in
archive_site_local:

See: #57718

See: #54800
See: ​https://github.com/macports/macports-ports/pull/3099

[Ryan Schmidt <git@…>]

In 38b25ceed71bf1753642740a97a2ab1f7f116fa6/macports-infrastructure (master):

Deploy nondistributable archives to private server

See: #54800

[Ryan Schmidt <git@…>]

In 59514c3fd771fdbe2a08c1e41e3a30bed05d362c/macports-infrastructure (master):

Also check archivesiteprivate when listing ports

See: #54800

[Ryan Schmidt <git@…>]

In ef98e89204113f876a3ff1ec80d43ad462432250/macports-infrastructure (master):

Use configured archive sites when installing

See: #54800

[Ryan Schmidt <git@…>]

In 1327a02e6cde03b4b1d6bb28115997941979786d/mpbb (master):

Also gather and upload nondistributable archives

See: #54800

[Ryan Schmidt <git@…>]

In 74d7a70f24dde533adf26cde09ee719f840baf42/mpbb (master):

Also check private archives when listing ports

See: #54800

[Ryan Schmidt <git@…>]

In 3994fd16b37de98ed3082ee36fdbed7cfe425d76/mpbb (master):

Also allow mpbb to download private archives

See: #54800

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Replying to ryandesign:

• On the buildbot server, set up a new directory outside of the rsync directory to store nondistributable archives.

Done. I've used a separate script to manually populate this directory with all current archives from all builders. After buildbot is modified to upload nondistributable archives there, I can run the separate script once more to sync everything up, then run another script to remove distributable archives from this directory, leaving only the nondistributable ones.

Done.

• Change mpbb so that it uploads nondistributable archives to there. (We don't need to worry about cleaning up old nondistributable archives; the existing cleanup script can probably be used for that.)

​https://github.com/macports/mpbb/pull/9

Buildbot configuration changes are also needed to supply the new arguments to mpbb.

​https://github.com/macports/macports-infrastructure/pull/8

Done.

Still to do: automating the renewal of the Let's Encrypt SSL certificates.

Not done yet.

• Modify Travis config to also find packages at the new CDN hostname by adding an entry to archive_sites.conf.

Still to do. Another possibility is setting archive_site_local in macports.conf.

Done.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Replying to ryandesign:

Still to do: automating the renewal of the Let's Encrypt SSL certificates.

Not done yet.

I guess automating renewal of the SSL certificates is the last thing to do here. I know there is a script running on Braeburn to automate the renewal of the certificates for the other hostnames on the CDN. Can we add packages-private.macports.org and distfiles-private.macports.org to that script? Or should I instead set up an instance of that script on my server? If so, where can I find the script?

[ryandesign (Ryan Carsten Schmidt)]

I've renewed the two certificates manually today.