Ports fail to fetch from Japanese ftp mirror

[a.lathrop@…]

I live in Japan.

When attempting to install/upgrade GNU packages/dependencies (ie. gsl, or gettext), I get:

---> Attempting to fetch gsl-1.14.tar.gz from ​ftp://ftp.dti.ad.jp/pub/GNU/gsl

and the program hangs there, without timing out. I can go the the ftp site via Firefox, and manually download the package to /opt/local/var/macports/distfiles, and retry, and it works fine. Something must be wrong with the ftp site, since other sources work fine, but it must also be a problem with macports, since Firefox is able to access the ftp site without issue.

[jmroot (Joshua Root)]

Are you using a proxy? Are you able to fetch this file using curl with FTP_PROXY set? Snow Leopard's version of libcurl has a bug related to handling of proxy environment variables ([rdar://problem/7997030], ​upstream), so you may need to disable proxy use for FTP.

[a.lathrop@…]

Not using a proxy.

[jochen@…]

Not sure why this was assigned to me, looks more like a base issue to me.

[jxzhu@…]

Replying to jmr@…:

Are you using a proxy? Are you able to fetch this file using curl with FTP_PROXY set? Snow Leopard's version of libcurl has a bug related to handling of proxy environment >variables ([rdar://problem/7997030], ​upstream), so you may need to disable >proxy use for FTP.

This should be related to the ticket (#25857) which was created by myself and is now closed. My system is behind the proxy. The Fetch via FTP always fails on my machine with Snow Leopard 10.6.4. But I don't have problem to open the ftp site by put the ftp link on a web browser.

Since I am relatively newer to MacPorts, I would appreciate if you can tell me in more detail How to fetch the file using curl with FTP_PROXY set? [I mean a list of procedures (commands) to get, e.g., the file

hdf5-1.8.5.tar.bz2 from ​ftp://ftp.hdfgroup.org/HDF5/current/src/ ].

Also how can I disable the proxy use for FTP?

Right now, I have in sudoers file

Defaults env_keep += "http_proxy HTTP_PROXY HTTPS_PROXY FTP_PROXY RSYNC_PROXY" Defaults env_keep += "ALL_PROXY NO_PROXY"

and in my .cshrc

setenv RSYNC_PROXY proxy.xxx.xxx:8080 #setenv FTP_PROXY proxy.xxx.xxx:8080

As you notice, I have commented out the line for FTP_PROXY environment variable. But it doesn't help.

Thanks.

[jmroot (Joshua Root)]

Can't reproduce; MacPorts fetches fine from this site for me.

[florian@…]

I can confirm and reproduce the ftp hang by setting up a local squid proxy (port install squid):

1) run tcpdump:

tcpdump -n -i lo0 host 127.0.0.1 and port 3128

2) in a 2nd terminal do this and observe tcpdump:

http_proxy=0.0.0.0:0
FTP_PROXY=127.0.0.1:3128
port fetch openconnect # this should hang

3) for comparison use curl:

FTP_PROXY=127.0.0.1:3128
curl -O ftp://ftp.infradead.org/pub/openconnect/openconnect-3.14.tar.gz # this works

This is the output of tcpdump:

A) port (2):

15:33:51.913101 IP 127.0.0.1.65288 > 127.0.0.1.3128: Flags [S], seq 3179921709, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862465 ecr 0,sackOK,eol], length 0
15:33:51.913163 IP 127.0.0.1.3128 > 127.0.0.1.65288: Flags [S.], seq 1404851936, ack 3179921710, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862465 ecr 114862465,sackOK,eol], length 0
15:33:51.913177 IP 127.0.0.1.65288 > 127.0.0.1.3128: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862465 ecr 114862465], length 0
15:33:51.913190 IP 127.0.0.1.3128 > 127.0.0.1.65288: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862465 ecr 114862465], length 0
# hangs here

B) curl (3):

15:33:12.245787 IP 127.0.0.1.65286 > 127.0.0.1.3128: Flags [S], seq 3483046350, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862069 ecr 0,sackOK,eol], length 0
15:33:12.245872 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [S.], seq 35148451, ack 3483046351, win 65535, options [mss 16344,nop,wscale 3,nop,nop,TS val 114862069 ecr 114862069,sackOK,eol], length 0
15:33:12.245888 IP 127.0.0.1.65286 > 127.0.0.1.3128: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 0
15:33:12.245902 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [.], ack 1, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 0

# until now same as above but continues:
15:33:12.246458 IP 127.0.0.1.65286 > 127.0.0.1.3128: Flags [P.], seq 1:271, ack 1, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 270
15:33:12.246494 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [.], ack 271, win 56855, options [nop,nop,TS val 114862069 ecr 114862069], length 0
15:33:12.246794 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [P.], seq 1:340, ack 271, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 339
15:33:12.246837 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [P.], seq 340:4436, ack 271, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 4096
15:33:12.246863 IP 127.0.0.1.3128 > 127.0.0.1.65286: Flags [P.], seq 4436:8532, ack 271, win 56888, options [nop,nop,TS val 114862069 ecr 114862069], length 4096
[...]

Could this be a libcurl bug?

---

It is impossible to explicitly disable/set a proxy in macports.conf if proxies are set in sys preferences. Can this option be added? E.g.:

proxy_ftp no

or

proxy_ftp disable

Instead of proxy_override_env these methods would be much better:

proxy_ftp env # use environment
proxy_ftp sys # use sys preferences
proxy_ftp host:port # set proxy here

Then we have better control of how a proxy is set.

Thanks!

[florian@…]

squid.conf for reproducing ftp hang with local squid proxy

[florian@…]

Here are some hints how to resolve this problem:

1) I traced down the problem to a buggy libcurl (7.19.7) which hangs on the call of select() in function Curl_socket_ready(). This is reproducible with the steps outlined in my previous post. This is the backtrace:

#0  0x00007fff85e9d932 in select$DARWIN_EXTSN ()
#1  0x0000000100068bbd in Curl_socket_ready ()
#2  0x000000010003a9f2 in ftp_easy_statemach (conn=0x100800408) at ftp.c:3056
#3  0x000000010003ad28 in ftp_connect (conn=0x100800408, done=0x7fff5fbfecee) at ftp.c:3186
#4  0x0000000100044cb5 in Curl_protocol_connect (conn=0x100800408, protocol_done=0x7fff5fbfecee) at url.c:3070
#5  0x0000000100047ea8 in setup_conn (conn=0x100800408, hostaddr=0x100115338, protocol_done=0x7fff5fbfecee) at url.c:4711
#6  0x0000000100048024 in Curl_connect (data=0x100803c08, in_connect=0x7fff5fbfed20, asyncp=0x7fff5fbfecef, protocol_done=0x7fff5fbfecee) at url.c:4787
#7  0x0000000100059d5f in connect_host (data=0x100803c08, conn=0x7fff5fbfed20) at transfer.c:2488
#8  0x0000000100059ff7 in Curl_perform (data=0x100803c08) at transfer.c:2625
#9  0x000000010005aa9f in curl_easy_perform (curl=0x100803c08) at easy.c:557
#10 0x000000010000b2ee in operate (config=0x7fff5fbff350, argc=3, argv=0x7fff5fbff6a8) at main.c:5002
#11 0x000000010000bcbc in main (argc=3, argv=0x7fff5fbff6a8) at main.c:5317

2) Snow Leopard comes with libcurl 7.19.7 so this fails:

http_proxy=0.0.0.0:0
FTP_PROXY=127.0.0.1:3128
/usr/bin/curl -v ftp://ftp.infradead.org/pub/openconnect/openconnect-3.15.tar.gz > /dev/null

3) The bug is apparently fixed in the libcurl that the curl port installs. So the macport installed curl works as expected:

http_proxy=0.0.0.0:0
FTP_PROXY=127.0.0.1:3128
/opt/local/bin/curl -v ftp://ftp.infradead.org/pub/openconnect/openconnect-3.15.tar.gz > /dev/null

4) port fetch calls curl fetch from Pextlib, however, Pextlib is linked against the buggy libcurl that comes with Snow Leopard:

otool -L /opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib
/opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib:
	Pextlib.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libcurl.4.dylib (compatibility version 6.0.0, current version 6.1.0)
	/usr/lib/libedit.2.dylib (compatibility version 2.0.0, current version 2.11.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

5) Conclusion: The easiest fix would be to link Pextlib against the libcurl (currently 7.23.1) from macports and not rely on a 2 years old libcurl that comes with the system. The current version includes lots of bug fixes and probably several known security vulnerabilities. So I see no point in using the old one.