livecheck should have curl request a compressed response

[ryandesign (Ryan Carsten Schmidt)]

Today I noticed that youtube-dl's livecheck was not matching anymore. The reason seems to be that, although we are not requesting one, GitHub is delivering a gzip-compressed response.

Arguably, it is a bug for GitHub to deliver a compressed response when one was not requested.

Arguably, it is a bug for curl not to decompress a compressed response it received.

Nevertheless, to avoid the problem, and also to save bandwidth, we should request a compressed response when doing livecheck. The command line flag for this is --compressed. The equivalent libcurl option is to set CURLOPT_ACCEPT_ENCODING to "". Note that before curl 7.21.6 this was called CURLOPT_ENCODING, and we should include code for that to support old versions of Mac OS X.

[ryandesign (Ryan Carsten Schmidt)]

$ curl -sO http://rg3.github.io/youtube-dl/download.html
$ file download.html 
download.html: gzip compressed data, from Unix
$ curl -sO --compressed http://rg3.github.io/youtube-dl/download.html
$ file download.html 
download.html: HTML document, UTF-8 Unicode text
$

[kurthindenburg (Kurt Hindenburg)]

Did something change since this ticket? I get "HTML document, UTF-8 Unicode text" for both now and livecheck appears to work.

[ryandesign (Ryan Carsten Schmidt)]

Yes; their servers have been intermittently returning correct or incorrect responses. They said they were trying out new CDN settings, which may have contributed to the problem.

Even if their problem is now resolved, we should request a compressed response for livecheck.

[dgilman (David Gilman)]

In 8c41ca7c752b29a6df48fc7ebf14b8a80aa43d57/macports-base (master):

Enable cURL's automatic decompression

Closes: #55011

[jmroot (Joshua Root)]

Is there any way to change this setting from a Portfile if it needs to fetch from a misconfigured server? I've seen plenty of cases where actual gzip files get served as though they were uncompressed files being transferred with gzip encoding.

[jmroot (Joshua Root)]

In fact, it would be safest to keep the old behaviour for everything other than livecheck.

[dgilman (David Gilman)]

I added a flag to the underling TCL command but I didn't think to propagate that to the livecheck. Can do a 2nd PR for that, and probably need to also add a flag to however the fetch phase works.

[jmroot (Joshua Root)]

Replying to dgilman:

Can do a 2nd PR for that, and probably need to also add a flag to however the fetch phase works.

Thanks, that would be good.

[tobypeterson]

Having a fair bit of trouble since I rebuilt with this change. Really need to verify that all ports fetch/checksum properly, this is a significant change in behavior.

[tobypeterson]

Notably, at least some GNU mirrors are exhibiting the behavior jmr mentioned.