rust @1.80.1_0: Some distfiles missing from *.distfiles.macports.org

[jmarshallidau]

• macOS 12.7.6 (x86_64)
• macOS 14.6.1 (arm64)

Over the past few days I've attempted and re-attempted upgrade from @1.78.0_0 on both platforms. Lots and lots of distfiles are fetched until this one. The list of download sites is exhausted and the final error is: Failed to fetch rust: The requested URL returned error: 404

The macOS systems are on an internal network behind a web proxy server and all have access to .distfiles.macports.org. The missing files are available on static.rust-lang.org but the macOS systems have no access to it. I could add that host to the relevant ACL on the proxy server but figure there must be a problem with whatever process is supposed to mirror these files to the distfiles.macports.org servers.

:notice:fetch --->  Attempting to fetch rust-std-1.79.0-x86_64-apple-darwin.tar.gz from http://aarnet.au.distfiles.macports.org/pub/macports/distfiles/cargo-crates
:debug:fetch Fetching distfile failed: The requested URL returned error: 404
:notice:fetch --->  Attempting to fetch rust-std-1.79.0-x86_64-apple-darwin.tar.gz from https://static.rust-lang.org/dist
:debug:fetch Fetching distfile failed: The requested URL returned error: 407
:notice:fetch --->  Attempting to fetch rust-std-1.79.0-x86_64-apple-darwin.tar.gz from https://distfiles.macports.org/cargo-crates
:debug:fetch Fetching distfile failed: The requested URL returned error: 404
...
:notice:fetch --->  Attempting to fetch rust-std-1.79.0-x86_64-apple-darwin.tar.gz from https://ywg.ca.distfiles.macports.org/mirror/macports/distfiles/cargo-crates
:debug:fetch Fetching distfile failed: The requested URL returned error: 404
:error:fetch Failed to fetch rust: The requested URL returned error: 404

[jmarshallidau]

Exactly the same problem after the port was updated to @1.81.0, except now it's failing to find rust-std-1.80.1-<blah> instead of rust-std-1.79.0-<blah>.

:info:fetch --->  rust-std-1.80.1-aarch64-apple-darwin.tar.gz does not exist in /opt/local/var/macports/distfiles/cargo-crates
:notice:fetch --->  Attempting to fetch rust-std-1.80.1-aarch64-apple-darwin.tar.gz from http://aarnet.au.distfiles.macports.org/pub/macports/distfiles/cargo-crates
:debug:fetch Fetching distfile failed: The requested URL returned error: 404
...
:notice:fetch --->  Attempting to fetch rust-std-1.80.1-aarch64-apple-darwin.tar.gz from https://ywg.ca.distfiles.macports.org/mirror/macports/distfiles/cargo-crates
:debug:fetch Fetching distfile failed: The requested URL returned error: 404
:error:fetch Failed to fetch rust: The requested URL returned error: 404

[ryandesign (Ryan Carsten Schmidt)]

Here's one of the latest server mirroring logs:

​https://build.macports.org/builders/jobs-mirror/builds/810561/steps/mirror/logs/stdio

There are some problems, including checksum mismatches that should be investigated, but I see no mention of any attempt to mirror rust-std-1.80.1-aarch64-apple-darwin.tar.gz. It does mention rust-std-1.77.0-aarch64-apple-darwin.tar.gz.

The machine that does the mirroring is x86_64. Normally the port would not have arm64 distfiles on an x86_64 machine but when using the +mirror_all_architectures variant, all distfiles for all architectures are added to the port. The mirroring process mirrors for all variants using the mirroring machine's OS version and architecture. The mirroring process also mirrors for each OS version and architecture using the default set of variants.

The rust port and the rust_build portgroup it includes are programmed to use rust version 1.81 (bootstrapping with rust 1.80) on macOS 10.13 and newer and rust version 1.78 (bootstrapping with 1.77) on macOS 10.12 and older. The machine that does the mirroring runs OS X 10.11. This is why the wrong version of the distfile is being mirrored.

I suggest the +mirror_all_architectures variant be expanded to also include all versions of the distfiles that might be needed by any OS version.

[jmarshallidau]

Now that the rust port has been upgraded to 1.82.0, it is the 1.81.0 files which cannot be found on the distfiles.macports.org servers. Fetched those files successfully from static.rust-lang.org.