Opened 8 months ago

Closed 8 months ago

#68563 closed defect (fixed)

intermittent 503 errors from {packages,distfiles}.macports.org

Reported by: barracuda156 Owned by: admin@…
Priority: Normal Milestone:
Component: server/hosting Version:
Keywords: Cc: contextnerror, chrstphrchvz (Christopher Chavez), rrze-ftp-admins@…
Port:

Description

See what is going on here, for example: ​https://github.com/macports/macports-ports/pull/21078 Also both R-related PRs.

Change History (23)

comment:1 Changed 8 months ago by jmroot (Joshua Root)

Component: buildbot/mpbb → server/hosting

Doesn't look like anything that could be fixed in the buildbot or mpbb code.

comment:2 in reply to:  1 Changed 8 months ago by barracuda156

Replying to jmroot:

Doesn't look like anything that could be fixed in the buildbot or mpbb code.

Got it, sorry. I just did not understand what happened :)

comment:3 Changed 8 months ago by jmroot (Joshua Root)

Summary: CI again broken → intermittent 503 errors from {packages,distfiles}.macports.org

comment:4 Changed 8 months ago by jmroot (Joshua Root)

In 9a735472646136ff02cffc9399fabe2cf8e6fa94/mpbb (master):

dependencies.tcl: attempt to mitigate 503 errors

Ensure that archivefetch is not attempted at all in install_dep_source,
since we only get there if it already failed. This should prevent
failure from archivefetch getting a 503 for a signature again.

This is at best a poor and partial workaround, since archivefetch
failing for ports that have an archive available in the first place
means that those ports are unnecessarily built from source, and
furthermore, fetching the distfiles might fail the same way.

See: #68563

comment:5 Changed 8 months ago by jmroot (Joshua Root)

Has anyone seen this outside of CI? It's possible it's an issue with GitHub's network, if they're using a proxy or something like that.

comment:6 Changed 8 months ago by kencu (Ken)

I have been reading that a number of projects (eg gmp) have been firewalling Microsoft’s servers lately due to massive overuse coming from GitHub.

If that might be related to this…

Last edited 8 months ago by kencu (Ken) (previous) (diff)

comment:7 Changed 8 months ago by jmroot (Joshua Root)

Unlikely to be related unless Fastly is doing the same. I do get this from https://www.macports.org/news/ at the moment: 

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<hr>
<address>Apache/2.4.38 (Debian) Server at www.macports.org Port 443</address>
</body></html>
Last edited 8 months ago by jmroot (Joshua Root) (previous) (diff)

comment:8 Changed 8 months ago by jmroot (Joshua Root)

According to the CDN stats, the significant numbers of 503s started late on the 24th and got much more frequent on subsequent days.

comment:9 Changed 8 months ago by markloch (Mark Lochridge)

EDIT: installing libmagic @1030AM PT 27OCT was successful, so apparently not repeatable ...

New user here, here's a repeatable case (on my part at least): install of libmagic consistently failing with error 503.

This first occurred while installing zeek. All other packages installed without issue. Installing libmagic separately yields the following, same results last night and this morning: 

% sudo ./port install libmagic
Password:
--->  Computing dependencies for libmagic
--->  Fetching archive for libmagic
Warning: Your DNS servers incorrectly claim to know the address of nonexistent hosts. This may cause checksum mismatches for some ports. See this page for more information: <https://trac.macports.org/wiki/MisbehavingServers>
--->  Attempting to fetch libmagic-5.45_0.darwin_22.x86_64.tbz2 from https://packages.macports.org/libmagic
--->  Attempting to fetch libmagic-5.45_0.darwin_22.x86_64.tbz2.rmd160 from https://packages.macports.org/libmagic
Error: Failed to archivefetch libmagic: Failed to fetch signature for archive: The requested URL returned error: 503
Error: See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_sysutils_file/libmagic/main.log for details.
Last edited 8 months ago by ryandesign (Ryan Carsten Schmidt) (previous) (diff)

comment:10 Changed 8 months ago by contextnerror

Cc: contextnerror added

comment:11 Changed 8 months ago by chrstphrchvz (Christopher Chavez)

Can the CI use a distfile/package mirror for now? ​https://github.com/macports/macports-ports/pull/21115

comment:12 Changed 8 months ago by chrstphrchvz (Christopher Chavez)

Cc: chrstphrchvz added

comment:13 in reply to:  11 Changed 8 months ago by aeiouaeiouaeiouaeiouaeiouaeiou

Now failed access to ftp.fau.de: ​https://github.com/macports/macports-ports/pull/21121/checks 

curl: (28) Failed to connect to ftp.fau.de port 443 after 75020 ms: Couldn't connect to server

comment:14 Changed 8 months ago by jmroot (Joshua Root)

If it's the origin server having problems, temporarily removing both it and the CDN from the list should work around the problem for everyone.

Last edited 8 months ago by jmroot (Joshua Root) (previous) (diff)

comment:15 Changed 8 months ago by jmroot (Joshua Root)

Cc: rrze-ftp-admins@… added

comment:16 Changed 8 months ago by jmroot (Joshua Root)

In b0871725a86ea018cc3d3917ab4c4fa0094142d8/macports-ports (master):

Temporarily disable primary mirror due to errors

See: #68563

comment:17 Changed 8 months ago by ryandesign (Ryan Carsten Schmidt)

mpbb is intentionally programmed to only use the main sites and not any mirrors:

​https://github.com/macports/mpbb/blob/9a735472646136ff02cffc9399fabe2cf8e6fa94/mpbb-checkout#L214

(mirror.fcix.net is an unintended exception to the rule because it does not use a macports.org hostname.)

This works fine for everything except the Apple Silicon build workers because they all live on the same network where local DNS rules map distfiles.macports.org to the private server.

The Apple Silicon build workers are at a different data center and will be talking to the real distfiles.macports.org.

comment:18 Changed 8 months ago by chrstphrchvz (Christopher Chavez)

In ad51a6cbcba31843010ad4f4eb6ca2ad65150824/macports-ports (master):

CI: temporarily prefer mirrors

See: #68563

comment:19 in reply to:  17 Changed 8 months ago by jmroot (Joshua Root)

Replying to ryandesign:

mpbb is intentionally programmed to only use the main sites and not any mirrors:

GitHub Actions is a slightly different situation again. It uses some mpbb commands but not mpbb checkout, but sets up a very similar macports.conf itself.

comment:20 Changed 8 months ago by jmroot (Joshua Root)

The FAU admins indicated by email that the problem seemed to be an unusual type of DDoS, and should now be resolved. CDN stats agree, so I'll revert the workarounds and continue to monitor for a bit.

comment:21 Changed 8 months ago by jmroot (Joshua Root)

In f9405fcf65e910f5589fb5b2310ae4bfb63ceb60/macports-ports (master):

Re-enable primary mirror

See: #68563

comment:22 Changed 8 months ago by jmroot (Joshua Root)

In 2a63d06ed613af5f16826167cfcda1f05817e283/macports-ports (master):

Revert "CI: temporarily prefer mirrors"

This reverts commit ad51a6cbcba31843010ad4f4eb6ca2ad65150824.

See: #68563

comment:23 Changed 8 months ago by jmroot (Joshua Root)

Resolution: → fixed
Status: new → closed
Note: See TracTickets for help on using tickets.