Point out when checksum errors are due to broken DNS or web servers

[ryandesign (Ryan Carsten Schmidt)]

We've had quite a lot of tickets filed lately about checksum errors, and I have a feeling a fair number of them are because of the "helpful" DNS servers ISPs seem to be so fond of providing these days. When asking such DNS servers about a hostname that does not exist, they do not respond that the hostname does not exist; instead, they respond with the IP address of the ISP's search page, and if you are trying to connect to it over HTTP, they do a web search for you and send you the results. OpenDNS does this by default, so does Time Warner's Roadrunner service and probably others. MacPorts can't currently recognize this, so it saves that search results page as the distfile, which then of course doesn't match the expected checksums.

The attached patch fixes this by using the "file" command to check if the downloaded file is an HTML page, and if so, discarding it and trying another mirror. I have tested this on Snow Leopard only. Before committing, I would like to verify that the "file" command on Leopard and preferably Tiger works the same way (i.e. accepts the same flags).

There is already special-case code in this procedure for dealing with SourceForge mirrors. Possibly the new code I propose could replace the old SourceForge code; I was unable to locate a test case that triggered the SourceForge special-case code.

[ryandesign (Ryan Carsten Schmidt)]

proposed patch

[ryandesign (Ryan Carsten Schmidt)]

The --mime-type flag appears to only be valid for the version of the file command on Snow Leopard (and the version in MacPorts); it doesn't exist in the version of file on Tiger or Leopard. But the --mime flag does, and it outputs the same information I need (and a little more that I don't need, which I can just discard).

[ryandesign (Ryan Carsten Schmidt)]

revised patch, tested on Tiger, Leopard, & Snow Leopard

[jmroot (Joshua Root)]

I really don't like the idea of legitimising the completely broken behaviour of these DNS servers (and web servers, since they must not even be serving the "helpful" page with a 404 code). The checksum failure reports do indicate a genuine issue with the ports (even if it would be an extremely minor one given working DNS). Giving additional information about the likely cause of a checksum failure would be OK.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmr@…:

I really don't like the idea of legitimising the completely broken behaviour of these DNS servers (and web servers, since they must not even be serving the "helpful" page with a 404 code).

Well, we could go on a campaign of lobbying the ISPs to fix this. I think this would be a lot of work and I don't know that it would be successful. Meanwhile the attached patch was a couple lines of code and is completely successful as far as I can tell in my testing.

The checksum failure reports do indicate a genuine issue with the ports (even if it would be an extremely minor one given working DNS). Giving additional information about the likely cause of a checksum failure would be OK.

The genuine issue is that the port is referring to a server that doesn't exist? True, but in the case of a properly-functioning DNS server, we simply skip it and move on; the patch in this ticket simply affords that same convenience to users with these increasingly-popular DNS servers. And the patch does print a detailed error message explaining that the distfile file was unexpectedly an HTML page, so if a port maintainer sees this output (either experienced themselves or submitted in a ticket) they can investigate then, just like they can if they see a message that a server could not be contacted.

For me, without the patch, I encounter the issue extremely often, because I often work at friends' houses and coffee shops, each of which generally has such a DNS server now. My mirror_sites.tcl on each of my local machines is modified to include each of the other machines (so they can fetch distfiles from each other quickly); when using my laptop elsewhere, these local machine names don't exist so I run into the issue for pretty much every port. So I would like to commit this patch.

[ryandesign (Ryan Carsten Schmidt)]

final version

[jmroot (Joshua Root)]

Let's be helpful without sweeping the problem under the rug. r70143

[ryandesign (Ryan Carsten Schmidt)]

Joshua also created wiki page MisbehavingServers describing the problem.

[ryandesign (Ryan Carsten Schmidt)]

See #33930 for the continuation.

[cooljeanius (Eric Gallager)]

Cc Me!