Proxy support broken by Leopard's sudo

[expiation@…]

New versions of Macports no longer respects the http_proxy environment variable. As there is no other way to set a proxy for Macports, this effectively means that the port command cannot be used behind a firewall using a proxy.

[juhnke123@…]

Hello,

same problem here but i found the following workaround. I modified the port script "/opt/local/bin/port" to include my proxies... Not a perfect solution but it works.

#!/bin/sh
#\
http_proxy=myproxy:8080
#\
ftp_proxy=myproxy:8080
#\
export http_proxy ftp_proxy
#\
exec /usr/bin/tclsh "$0" "$@"
# port.tcl
...

[kballard (Lily Ballard)]

If you edit your /opt/local/etc/macports/macports.conf to not strip http_proxy (look at the end of the file), then it should work fine. The only other problem is sudo also likes to sanitize the environment.

I have a report of MacPorts 1.6 working with the following command to use an http proxy:

sudo env http_proxy=http://proxy.url.here port install someport

However, we really should support proxies natively.

[jmroot (Joshua Root)]

Perhaps we should ship macports.conf with:

extra_env               http_proxy HTTPS_PROXY FTP_PROXY ALL_PROXY NO_PROXY

[blb@…]

This support has actually been there for some time, and it still appears to work fine in my quick testing. Note that as jmr mentions in comment 5 that only the http_proxy is lower case, all the others are upper case (see 'man curl' for all the env vars applicable).

I was able to 'port fetch' and it went through my proxy (set with http_proxy); I don't have extra_env set at all. Also, for sudo, this shouldn't be stripped either; try a 'sudo env |grep -i proxy' to find out for sure. It definitely isn't stripped here:

sudo port -dv fetch
...
--->  Attempting to fetch bzip2-1.0.4.tar.gz from http://www.bzip.org/1.0.4/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  821k  100  821k    0     0  6643k      0 --:--:-- --:--:-- --:--:-- 9289k

That came through the proxy since there's no way my DSL connection is that fast...

[jmroot (Joshua Root)]

Maybe listing the proxy variables in extra_env isn't necessary, but on my system (Leopard), MacPorts definitely wasn't using my proxy before I added the following to /etc/sudoers:

Defaults        env_keep += "http_proxy HTTPS_PROXY FTP_PROXY"
Defaults        env_keep += "ALL_PROXY NO_PROXY"

[blb@…]

Interesting; if you comment those out then run sudo sudo -V does it show anything proxy-related being filtered out?

[jmroot (Joshua Root)]

It filters out almost everything that isn't explicitly listed with env_keep, since it has this first:

Defaults        env_reset

[blb@…]

Ah, I see the problem now; I upgraded from 10.4 to 10.5 and it kept my older sudoers; apparently a new 10.5 sudoers does specify the env_reset, so I guess that the sudo env ... trick is now a necessity with 10.5. Unless we want to tell people to do as you say with env_keep in sudoers...

[smith.kennedy macosforge@…]

A nice variation or alternative would be to update the port command to do something like "scutil --proxy" and parse the results so that it would automagically use the settings made in the Network prefs panel, which may vary according to location (which some of us laptop users use).

[smith.kennedy@…]

Follwoing up from my comment yesterday (annoying problems with Trac and WordPress and my account email having a "+macosforge@…" suffix that I just resolved) I tried adding lines to the top of /opt/local/bin/port like so, and it seems to work in locations with and without HTTP proxies set (I couldn't put the statements on separate lines - tclsh seems to get angry and bail out when I format it in a more nice manner):

#!/bin/sh
# -*- coding: utf-8; mode: tcl; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- vim:fenc=utf-8:filetype=tcl:et:sw=4:ts=4:sts=4 \
if [ `scutil --proxy | grep HTTPEnable | cut -d : -f 2 | cut -c 2-` -eq 1 ]; then http_proxy=`scutil --proxy | grep HTTPProxy | cut -d : -f 2 | cut -c 2-`:`scutil --proxy | grep HTTPPort | cut -d : -f 2 | cut -c 2-`; export http_proxy; fi
#\
if [ `scutil --proxy | grep FTPEnable | cut -d : -f 2 | cut -c 2-` -eq 1 ]; then http_proxy=`scutil --proxy | grep FTPProxy | cut -d : -f 2 | cut -c 2-`:`scutil --proxy | grep FTPPort | cut -d : -f 2 | cut -c 2-`; export FTP_PROXY; fi
#\
exec /usr/bin/tclsh "$0" "$@"
# port.tcl

It seems to work on Mac OS 10.4 and 10.5 - not sure about earlier versions but I will check on a test system tomorrow. I tried implementing it in tcl, but I haven't ever written tcl before, and my efforts to set environment variables in a tcl script failed. Here is my failed (and likely very ugly) attempt:

#Set proxies according to System Configuration Framework settings
set http_proxy ""
set FTP_PROXY ""
set http_proxy_enabled [exec scutil --proxy | grep HTTPEnable | cut -d : -f 2 | cut -c 2-]
if { $http_proxy_enabled == 1 } {
    set http_proxy [exec scutil --proxy | grep HTTPProxy | cut -d : -f 2 | cut -c 2-]:[exec scutil --proxy | grep HTTPPort | cut -d : -f 2 | cut -c 2-]
    set http_proxy
}
set ftp_proxy_enabled [exec scutil --proxy | grep FTPEnable | cut -d : -f 2 | cut -c 2-]
if { $ftp_proxy_enabled == 1 } {
    set FTP_PROXY [exec scutil --proxy | grep FTPProxy | cut -d : -f 2 | cut -c 2-]:[exec scutil --proxy | grep FTPPort | cut -d : -f 2 | cut -c 2-]
}
puts "http_proxy = $http_proxy"
puts "FTP_PROXY = $FTP_PROXY"

set env("http_proxy") {$http_proxy}
set env("FTP_PROXY") {$FTP_PROXY}

Any patches in this vein would be appreciated, as I prefer to not have to maintain patched copies of software (if I did, I wouldn't bother reporting my changes...)

[blb@…]

Parsing the output from scutil is fragile and could run into issues. The best way would be to integrate what scutil does into MacPorts and set the env vars from there. I will attach a patch (rooted in trunk/) which adds a new command to MacPorts which updates the HTTP, HTTPS, and FTP proxy settings from SystemConfiguration (if they aren't already set, otherwise it leaves them alone). It also takes care of NO_PROXY. It should do nothing at all on non-Mac systems.

[blb@…]

Patch (from trunk/) to add better libcurl proxy support from SystemConfiguration

[jmroot (Joshua Root)]

I don't suppose that works if a PAC file is being used? We probably need to be able to get proxy settings from macports.conf as well.

[blb@…]

Replying to jmr@macports.org:

I don't suppose that works if a PAC file is being used? We probably need to be able to get proxy settings from macports.conf as well.

Nope, not from a PAC since we'd need to be able to execute it in that case; I don't think we want to embed a Javascript interpreter inside MacPorts...

So yeah, a fallback with some statically-defined entries in macports.conf would probably be best since env vars alone won't cut it on 10.5.

[smith.kennedy@…]

blb, thanks for the patch and the more appropriate implementation!

[blb@…]

Alright, time for a better patch. This one allows macports.conf and SystemConfiguration to set proxy information, as needed. Also, since it's dealing with proxy information, rsync support for it was also added.

Basically, it sets the pertinent environment variables during mportinit (http_proxy, HTTP_PROXY, etc) as needed. The stuff added to macports.conf.in discuss precedence information.

[blb@…]

diff (against trunk/base as of r37802) to add better proxy support

[smith.kennedy@…]

Cc Me!

[blb@…]

De-bitrotted and committed in r47194.