Opened 4 years ago
Closed 4 years ago
#60558 closed defect (fixed)
mirror_sites.tcl: trimming and updating
Reported by: | iefdev (Eric F) | Owned by: | Eric F <3958664+iefdev@…> |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | base | Version: | 2.6.2 |
Keywords: | Cc: | ryandesign (Ryan Carsten Schmidt), kencu (Ken) | |
Port: |
Description
As requested: A separate ticket for trimming and updating the mirror_sites.tcl
Continues the discussion from: #60509
Including the last parts/replies:
Replying to ryandesign:
Speaking of the list. Except from trimming it, I think it needs an update. For example…
- the PHP mirrors are gone, aren't they? All redirects to www.php.net now, and there's no mirror.php page anymore.
- For debian, they also have their default one: http://ftp.debian.org/debian/pool/main/ (ie without a country code in it).
- I also saw a dead link. There's no unix section there anymore: http://ftp.sunet.se/pub/
<snip />
And, most mirrors use https now, don't they? Same example: http://ftp.sunet.se/pub/ -> https://ftp.sunet.se/pub/ When/if trimming the list – https sites should be prioritized. Incase some of them uses redirects (->https) it could also make it faster.
ryandesign (Ryan Schmidt):
Please file separate tickets for the issues with the mirror list ...
kencu (Ken):
all the https links will likely fail on older systems about 10.9 and less unless users have followed my bootstrap procedure.
ryandesign (Ryan Schmidt):
Not necessarily.
Change History (5)
comment:1 Changed 4 years ago by iEFdev
comment:2 follow-ups: 3 4 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
Different servers implement different requirements for https, by virtue of the list of SSL/TLS protocols they support, the list of encryption algorithms they support, and/or whether the certificate authority through which their certificate is issued is recognized by the system. GitHub and SourceForge for example both implemented requirements a few years ago that surpass the abilities of /usr/bin/curl on OS X 10.8 and earlier, but not all servers are that restrictive, and some are more restrictive than that.
The distfiles_scheme
and packages_scheme
variables aren't great, because they presume that the decision about whether to use http or https is always the same on an OS version. It is not; it varies by server. So there should really be separate variables for each server, for those that offer http and https, if that https is not compatible with all systems.
We use these variables only for official MacPorts mirrors. For other servers, if they offer https, we have been switching unconditionally to https in the interest of privacy, even if that https does not support all OS versions. In that case older OS versions may try to connect to those servers, fail, and then download successfully from another server, such as our mirrors; that's fine. Figuring out which OS versions support https on each of hundreds of other servers would be too tedious.
As for the other issues, thanks for pointing them out. I wasn't aware of them. You're correct for example that the php mirrors are gone and we need to adapt MacPorts accordingly. Regarding the dead link, it's inevitable that servers will go way or that they will stop hosting some content. There is no automated process in place to keep mirror_sites.tcl up to date; it requires someone like you to notice a problem and for it to be fixed manually. If you'd like to submit a PR to fix any or all of this that would be great, otherwise I'm sure someone will get to it eventually.
comment:3 Changed 4 years ago by iEFdev
Replying to ryandesign:
As for the other issues, thanks for pointing them out. I wasn't aware of them. You're correct for example that the php mirrors are gone and we need to adapt MacPorts accordingly. Regarding the dead link, it's inevitable that servers will go way or that they will stop hosting some content. There is no automated process in place to keep mirror_sites.tcl up to date; it requires someone like you to notice a problem and for it to be fixed manually. If you'd like to submit a PR to fix any or all of this that would be great, otherwise I'm sure someone will get to it eventually.
So, strip down the mirrors to just one (php).
Just made a quick test to see if one could do a status test:
$ for site in $(sed -n '18,30p' mirror_sites.tcl); do \ > echo -n "${site}: " && curl -I ${site} 2>/dev/null | grep HTTP; done https://mirror.aarnet.edu.au/pub/apache/: HTTP/1.1 200 OK https://archive.apache.org/dist/: HTTP/1.1 200 OK https://www.apache.org/dist/: HTTP/1.1 302 Found http://mirror.cc.columbia.edu/pub/software/apache/: HTTP/1.1 200 OK http://mirror.facebook.net/apache/: HTTP/1.1 200 OK http://www.gtlib.gatech.edu/pub/apache/: HTTP/1.1 200 OK http://mirrors.ibiblio.org/apache/: HTTP/1.1 200 OK ftp://ftp.infoscience.co.jp/pub/net/apache/dist/: <--- err/timeout http://mirror.internode.on.net/pub/apache/: HTTP/1.1 200 OK http://apache.is.co.za/: HTTP/1.1 200 OK https://www.mirrorservice.org/sites/ftp.apache.org/: HTTP/1.1 200 OK http://apache.pesat.net.id/: HTTP/1.1 403 Forbidden http://apache.mirror.rafal.ca/: HTTP/1.1 200 OK
A timeout err doesn't tell if the site's gone or not. If not obvious, perhaps better to keep them?
—
And it looks like mySQL have removed their mirror page as well. Couldn't find any info about it though. Tested a random few mirrors… and ~ half are gone. I don't know what to make of that. I can remove the bad ones, but there's no way to generate a new/updated list.
When clicking on a download link:
https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.20.zip ---> https://cdn.mysql.com//Downloads/MySQL-8.0/mysql-8.0.20.zip (via Finder info)
So, maybe just use https://dev.mysql.com/get/Downloads/
as a single mirror then?
Well, I just wanted to take a quick look. Another day.
comment:4 Changed 4 years ago by iEFdev
Replying to ryandesign:
There is no automated process in place to keep mirror_sites.tcl up to date; it requires someone like you to notice a problem and for it to be fixed manually. If you'd like to submit a PR to fix any or all of this that would be great, otherwise I'm sure someone will get to it eventually.
Ok, I made this one: https://github.com/macports/macports-ports/pull/7257
comment:5 Changed 4 years ago by Eric F <3958664+iefdev@…>
Owner: | set to Eric F <3958664+iefdev@…> |
---|---|
Resolution: | → fixed |
Status: | new → closed |
Replying to kencu (Ken):
Could
${distfiles_scheme}://
be utilized (more) for that? Like here.