archive_site_local URLs are attempted twice

[ryandesign (Ryan Carsten Schmidt)]

With archive_site_local's basic functionality fixed (#57717), we now see that the specified site is tried twice. With macports.conf containing:

archive_site_local https://packages-private.macports.org/:tbz2

We see:

--->  Fetching archive for libpng
DEBUG: Executing org.macports.archivefetch (libpng)
DEBUG: euid/egid changed to: 0/0
DEBUG: chowned /opt/local/var/macports/incoming to macports
DEBUG: euid/egid changed to: 502/501
--->  libpng-1.6.36_0.darwin_15.x86_64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
--->  Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https://packages-private.macports.org/libpng
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not Found
--->  Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https://packages-private.macports.org/libpng
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not Found
--->  Attempting to fetch libpng-1.6.36_0.darwin_15.x86_64.tbz2 from https://packages.macports.org/libpng
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed: The requested URL returned error: 404 Not Found

portfetch::checksites adds each URL twice: once for use with any tag, and then again for the specific tag. Does it need to do that? If so, could the list be deduplicated, so that only one copy of each URL remains? Would the end of portfetch::checksites be a good place to do that deduplication, or is it better done elsewhere?

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[ryandesign (Ryan Carsten Schmidt)]

In 847208cf7a361d2ed6dc5f4f03184068690958d2/macports-ports (master):

Don't add packages.macports.org to archive_site_local

packages.macports.org is already in the default archive sites and each
site in archive_site_local gets added to the list twice. Since
packages.macports.org now has a lower pingtime than packages-private,
MacPorts tries three times to download a nondistributable package from
packages.macports.org without trying packages-private at all.

See: #57719
See: #57720
See: ​https://lists.macports.org/pipermail/macports-dev/2021-April/043259.html

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

portfetch::checksites adds each URL twice: once for use with any tag, and then again for the specific tag. Does it need to do that?

After some debugging, I think I now understand that the local sites are added without a tag at the top of checksites (for distfiles that don't specify a tag) and then, at the end, are added again for every tag specified in any URL added up to that point (for distfiles that do specify a tag). The problem is that archive_site_local URLs must end with :tbz2 (or similar), which looks like a tag, so MacPorts adds the local URLs a second time for that tag. So I'm trying to find a way to restructure things to fix that.

checksites is confusing because of the number of different things it does; because it does not return anything but sets global variables; and because its preceding comment block is hard to parse.

I need to see if we already have tests for checksites. If not, I should write some to cover existing functionality and to verify the fix for this issue.

If so, could the list be deduplicated, so that only one copy of each URL remains? Would the end of portfetch::checksites be a good place to do that deduplication, or is it better done elsewhere?

Aside from any fix for the preceding, deduplication might be a good idea anyway. I'm investigating doing this as part of this fix, perhaps using a ​dict; this code was probably written back when we still had to be compatible with earlier versions of Tcl that didn't have dict.