preferred_hosts has no effect on archive_site_local

[ryandesign (Ryan Carsten Schmidt)]

With archive_site_local's basic functionality fixed (#57717), we now see that preferred_hosts has no effect on archive_site_local. With macports.conf containing:

archive_site_local https://packages-private.macports.org/:tbz2
preferred_hosts packages.macports.org

We see that, contrary to intentions, nevertheless packages-private.macports.org was tried before packages.macports.org:

--->  Fetching archive for libpng
DEBUG: Executing org.macports.archivefetch (libpng)
DEBUG: euid/egid changed to: 0/0
DEBUG: chowned /opt/local/var/macports/incoming to macports
DEBUG: euid/egid changed to: 502/501
--->  libpng-1.6.36_0.darwin_15.x86_64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
--->  Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https://packages-private.macports.org/libpng
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not Found
--->  Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https://packages-private.macports.org/libpng
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not Found
--->  Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https://packages.macports.org/libpng
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not Found

[ryandesign (Ryan Carsten Schmidt)]

In 531720bba7af24204b507f26bdefbc60e58556c0/macports-ports (master):

Travis: Prefer public archives over private ones

I had tried to do this with preferred_hosts before, but it appears to
have no effect on archive_site_local.

See: #57720

Instead list both the public and the private URLs in archive_site_local.
They appear to be tried in the order listed.

But then we need to fix the base bug with multiple URLs in
archive_site_local:

See: #57718

See: #54800
See: ​https://github.com/macports/macports-ports/pull/3099

[jmroot (Joshua Root)]

This is working as intended. The *_local variables are for a single URL that is, as the name implies, local and thus expected to have the best performance.

[ryandesign (Ryan Carsten Schmidt)]

I can understand that point of view. However, what we need for Travis CI and our Buildbot setup is to be able to communicate to MacPorts that we want one URL checked first, a different URL checked second, and no other URLs checked. archive_site_local seemed like a way to do that, but it's becoming frustrating when the behavior is not defined/documented anywhere and seems to change at random when I move to a different computer for example.

[ryandesign (Ryan Carsten Schmidt)]

I originally thought that multiple URLs in archive_site_local were tried in the order specified, based on some initial testing. But my test wasn't realistic, because I used the same hostname (but a different path) for each URL. In that case, they are indeed tried in order, as far as I can tell.

It appears that these hosts are still pinged; times for these hosts are recorded in the pingtimes file. If we're still pinging them and trying them in ping order, it should be allowed to specify one or more that are preferred, using preferred_hosts.

However I cannot determine whether we are actually trying them in ping order, or in the order specified, or in random order. It seems to change every time I test it.

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[ryandesign (Ryan Carsten Schmidt)]

In 847208cf7a361d2ed6dc5f4f03184068690958d2/macports-ports (master):

Don't add packages.macports.org to archive_site_local

packages.macports.org is already in the default archive sites and each
site in archive_site_local gets added to the list twice. Since
packages.macports.org now has a lower pingtime than packages-private,
MacPorts tries three times to download a nondistributable package from
packages.macports.org without trying packages-private at all.

See: #57719
See: #57720
See: ​https://lists.macports.org/pipermail/macports-dev/2021-April/043259.html

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed