wget @1.19.4_2 breaks Digest Authentication

[jak574]

Updating from wget 1.19.4_0 to 1.19.4_2 broke Digest Authentication. It now errors out with "Unknown authentication scheme."

Example output from wget 1.19.4_2:

%> wget -O -o --no-check-certificate https://www.redacted.com/~redacted
--2018-03-16 12:14:04--  https://www.redacted.com/~redacted
Resolving www.redacted.com (www.redacted.com)... 123.123.123.123
Connecting to www.redacted.com (www.redacted.com)|123.123.123.123|:443... connected.
WARNING: cannot verify www.redacted.com's certificate, issued by â:
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 401 Authorization Required
Unknown authentication scheme.

Username/Password Authentication Failed.

Before (using wget 1.19.4_0):

%>wget -O -o --no-check-certificate https://www.redacted.com/~redacted
--2018-03-16 12:14:59--  https://www.redacted.com/~redacted
Resolving www.redacted.com (www.redacted.com)... 123.123.123.123
Connecting to www.redacted.com (www.redacted.com)|123.123.123.123|:443... connected.
WARNING: The certificate of â is not trusted.
WARNING: The certificate of â hasn't got a known issuer.
HTTP request sent, awaiting response... 401 Authorization Required
Authentication selected: Digest realm="Redacted Restricted", nonce="8n7z6IlnBQA=7481515c6fgdgfda85a52fb534bf11af6ab318f", algorithm=MD5, domain="https://redacted/ https://www.redacted.com/ http://www.redacted.com/ http://www.redacted.com http://redacted/ http://www.redacted.com/~redacted/", qop="auth"
Connecting to www.redacted.com (www.redacted.com)|123.123.123.123|:443... connected.
WARNING: The certificate of â is not trusted.
WARNING: The certificate of â hasn't got a known issuer.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.redacted.com/~redacted/ [following]
--2018-03-16 12:15:00--  https://www.redacted.com/~redacted/
Connecting to www.redacted.com (www.redacted.com)|123.123.123.123|:443... connected.
WARNING: The certificate of â is not trusted.
WARNING: The certificate of â hasn't got a known issuer.
HTTP request sent, awaiting response... 401 Authorization Required
Authentication selected: Digest realm="Redacted Restricted", nonce="EPv06IlnBQA=8c2fa442a625c6bdfgdfgdf8f36853ec45573b029", algorithm=MD5, domain="https://redacted/ https://www.redacted.com/ http://www.redacted.com/ http://www.redacted.com http://redacted/ http://www.redacted.com/~redacted/", qop="auth"
Connecting to www.redacted.com (www.redacted.com)|123.123.123.123|:443... connected.
WARNING: The certificate of â is not trusted.
WARNING: The certificate of â hasn't got a known issuer.
HTTP request sent, awaiting response... 200 OK
Length: 7715 (7.5K) [text/html]
Saving to: â

-o                                 100%[================================================================>]   7.53K  --.-KB/s    in 0s      

2018-03-16 12:15:00 (273 MB/s) - â saved [7715/7715]

[jak574]

Offending change seems to have been made here:

changeset:126f8d18bf1c975c4f1dd32430c324bf770c3e4d/macports-ports

[ryandesign (Ryan Carsten Schmidt)]

Replying to jak574:

Offending change seems to have been made here:

changeset:126f8d18bf1c975c4f1dd32430c324bf770c3e4d/macports-ports

That change (to 1.19.4_1) should not have changed anything. It should merely have made explicit what was already happening.

The relevant change is probably the next one (to 1.19.4_2) which changed the ssl variant from using gnutls to using openssl:

changeset:886e416e16c99243c931d2a1de65cb7a2a413e71/macports-ports

To verify that that's the cause, you could try reinstalling the port with the newly-added gnutls variant and see if it works again:

sudo port install wget +gnutls

[jak574]

I tried it with both +ssl and +gnutls and digest authentication doesn't work either way with the 1.19.4_2 portfile. Both versions error out with "Unknown authentication scheme".

The git check-in I pointed to above specifically adds the following line to the configuration:

--disable-digest

Unsurprisingly, that disables digest authentication support. The old portfile didn't have that and digest worked. I modified the port file to change this line to

--enable-digest

it fixed the issue.

So rather than making explicit what was already happening, I believe that the update to the port file is explicitly turning off features that would otherwise have been on by default.

I've been using wget in a production system with MacPorts for about 10 years in this configuration. The upgrade to 1.19.4_2 broke that system.

[jak574]

I just took a quick look at the wget configure file. --disable-digest is considered an "Optional Feature". I.e. if you don't specifically add it to the configure command, then digest is enabled by default.

[ryandesign (Ryan Carsten Schmidt)]

Ok, thanks for checking. ​The intention of the first commit was to make explicit what was already happening implicitly. Looks like we inadvertently turned off an option that we didn't know was on by default. We should turn it back on.

Looking closer at that commit, and at the difference in configure output before and after the change, I see that the following features which were on before are now off:

:info:configure   Digest:            no
:info:configure   NTLM:              no
:info:configure   OPIE:              no
:info:configure   POSIX xattr:       no
:info:configure   Debugging:         no

I think we should turn them all on again. Jan?

[pmetzger (Perry E. Metzger)]

+1 for turning them back on again.

[janstary (Jan Starý)]

If people are using them, we surely should. Fixed in ​https://github.com/macports/macports-ports/pull/1440

[janstary (Jan Starý)]

NTLM is Windows NT Lan Manager ​https://en.wikipedia.org/wiki/NT_LAN_Manager - do we want that too?

[pmetzger (Perry E. Metzger)]

I merged Jan's pull request.