Slow mirror: nue.de.packages.macports.org

[agraef (Albert Graef)]

Ever since I started using MacPorts, I've been annoyed by the consistently abysmal download rates offered by this server. Apparently MP uses it by default in Germany (presumably because of ping times), but I've blacklisted it now.

This server usually starts at download rates in the 60K/s ballpark, but quickly drops down to some 20K/s, and frequently stalls as well. I have a reasonably fast DSL connection offering some 3 MB/sec, so I'm pretty sure that it's on their side. Might be an issue with high workloads, but I doubt it as I rarely get better rates there, no matter what the time of day or night is. This is completely unacceptable; it makes upgrades basically impossible.

The server in the UK works quite well in comparison, although it isn't always what I consider fast in this day and age either. packages.macports.org (which I guess is the central server?) usually works best for me.

I don't expect anyone to act on this, but I wanted to document this issue at least, so that other German users experiencing the same issues know that they are not alone, and can help themselves by blacklisting the server in their macports.conf file.

Also, in case the providers of nue.de.packages.macports.org read this, it would be interesting to hear your comments and whether the issue may actually lie elsewhere, or whether you see a way of fixing it. I'd happily use a mirror in Germany if there was a usable one.

[larryv (Lawrence Velázquez)]

The list of mirrors provides admin contact information.

[agraef (Albert Graef)]

Ok, thanks for pointing that out. I see that you cc'ed that address already, so hopefully they will be able to comment. Maybe it's just some kind of misconfiguration. I know that the servers there offer plenty of other mirrors (I've probably used some of them before), and as a university they ought to have a direct link to the European backbone, so they should be able to do better. Otherwise I can ask our computing center at the JGU in Mainz whether they might be able to provide a mirror.

[agraef (Albert Graef)]

It gets weirder. Just tried downloading a bigger file (some 20 MB) from there directly using wget and got some 20-30K/s again. Did the same on my Linux box sitting right beside my MacBook and, lo and behold, the file came down in just a few seconds, giving me the kind of download rates I'd expect with the connection I have. This is reproducible, and it's only on their site. Big files from bitbucket.org download on the Mac just fine.

So the problem clearly is with nue.de.packages.macports.org, and only if the client is a Mac. Dear rrze-ftp-admins@…, now I'd *really* like to know what explanation for that kind of behavior is. Throttling down Macs when they access a MacPorts mirror? Come on. Are you that short on bandwidth? ;-)

[neverpanic (Clemens Lang)]

I cannot reproduce your problems. In fact, I made nue.de.*.macports.org my default mirror, because it is very fast. For example:

--->  Fetching distfiles for qt4-mac
--->  Attempting to fetch qt-everywhere-opensource-src-4.8.6.tar.gz from http://nue.de.distfiles.macports.org/macports/distfiles/qt4-mac
      [                                        ]  45.3 %        speed: 3.1 MiB/s
^C

and that's just because the connection here isn't any faster. Using my server yields similar results:

:) clemens@towel:~$ curl -L http://nue.de.distfiles.macports.org/macports/distfiles/qt4-mac/qt-everywhere-opensource-src-4.8.6.tar.gz >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  230M  100  230M    0     0  10.3M      0  0:00:22  0:00:22 --:--:-- 10.3M

That's pretty much as good as it gets on a 100M connection.

As an FAU alumni I know that Erlangen is part of the backbone of the German research network (DFN).

It gets weirder. Just tried downloading a bigger file (some 20 MB) from there directly using wget and got some 20-30K/s again. Did the same on my Linux box sitting right beside my MacBook and, lo and behold, the file came down in just a few seconds, giving me the kind of download rates I'd expect with the connection I have. This is reproducible, and it's only on their site. Big files from bitbucket.org download on the Mac just fine.

That makes me think the problem is actually with your local machine or network configuration, rather than with the server. Wifi vs. non-wifi, maybe?

Are you that short on bandwidth? ;-)

Nope. They mirror MacPorts so they can wast..., I mean use more bandwidth.

[neverpanic (Clemens Lang)]

Relaying an answer from the admins, since they don't have a trac account:

It probably is some kind of misconfiguration, but not on the server side. I cannot reproduce these horrible transfer rates from any network connection available to me. The server is in no way close to its limits, neither connectionwise nor I/O. As you correctly stated, we are directly connected to the DFN (german scientific network) which usually has rather good connectivity throughout Europe, and Erlangen even is one of the four supercores around which that network is built.

Also note that we do publish quite a lot of statistics about the mirrors we run, including an average transfer rate per HTTP connection. While that isn't really exact, more of a guesstimate, the current ATR for the macports mirror isn't even close to the numbers reported: ​https://ftp.fau.de/cgi-bin/show-ftp-stats.cgi?statstype=1&datum=2014-10-09&orderby=mirrorname&orderdir=asc

My first suggestion would be to check whether the connection goes through IPv4 or IPv6, and try the respective other. There are providers that seriously mess up their IPv6 connectivity (and thanks to "happy eyeballs" in webbrowsers customers hardly notice), so that would be an explanation that would not surprise me.

[agraef (Albert Graef)]

Replying to cal@…:

I cannot reproduce your problems. In fact, I made nue.de.*.macports.org my default mirror, because it is very fast.

Thanks, that's the kind of valuable information I've been looking for. I've been wondering whether I'm the only one who is affected, so I'll have to debug it on my side.

As an FAU alumni I know that Erlangen is part of the backbone of the German research network (DFN).

That's what I expected. And I can see on my other machines in the same home network that your servers are fast. Therefore I'm baffled that they're so slow on my Mac. And I've been seeing this consistently since I started using MacPorts.

That makes me think the problem is actually with your local machine or network configuration, rather than with the server. Wifi vs. non-wifi, maybe?

Nope, I tried both. Even on the same wire as my Lenovo laptop running Linux, which exhibits none of these issues. Also, other sites work fine on the Mac. speedtest.net shows some 23 Mbits/s download and 2 Mbits/s upload speed, which matches what my ISP offers very closely. So I can't imagine that it's an issue with my Fritz!Box (which is fairly new and running the latest firmware) or the ISP either.

The MacBook is running the latest Mavericks, is fully updated and not running any unusual software which might degrade network performance either, except maybe for BitTorrent Sync and Dropbox, but most of the time these just sit idle not using any network bandwidth.

If anyone has another idea what the issue with my Mac might be or what else I may try, please let me know. I'm not exactly a noob (been using various Un*x systems since the mid 1980s and Linux since 1993), but I'm not much of a network expert, so any advice will be appreciated.

Nope. They mirror MacPorts so they can wast..., I mean use more bandwidth.

Ha. :)

[neverpanic (Clemens Lang)]

Have you looked into IPv4 vs. IPv6? Your speedtest might use IPv4 and be perfectly fine while the download from the macports mirror uses IPv6 and is slow (e.g., via a tunnel or something; Windows machines have been known to advertise their Toredo tunnels on local networks, causing problems like these).

[agraef (Albert Graef)]

Replying to cal@…:

Have you looked into IPv4 vs. IPv6? Your speedtest might use IPv4 and be perfectly fine while the download from the macports mirror uses IPv6 and is slow (e.g., via a tunnel or something; Windows machines have been known to advertise their Toredo tunnels on local networks, causing problems like these).

There's no Windows server on my network, and the network preferences on the Mac show nothing unusual (cf. ​https://www.dropbox.com/s/i1e2vn4iasuw873/Screenshot%202014-10-12%2014.37.48.png?dl=0). IPv6 is set to automatic and both IPv4 and IPv6 addresses seem to be configured all right through DHCP.

Can anyone suggest a way to check whether the connection actually goes through IPv4 or IPv6? I must admit that I don't know how to do this.

One weird thing I noticed now is that the download rate seems to depend on which file I'm trying to download. With ​http://nue.de.packages.macports.org/macports/packages/gtk3/gtk3-3.10.0_0+x11.darwin_10.x86_64.tbz2 it soon drops down to a crawl, while with the file you tried it works ok (not blazingly fast, but workable).

curl -L http://nue.de.packages.macports.org/macports/packages/gtk3/gtk3-3.10.0_0+x11.darwin_10.x86_64.tbz2 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 46 13.2M   46 6271k    0     0  91321      0  0:02:32  0:01:10  0:01:22 23372

curl -L http://nue.de.distfiles.macports.org/macports/distfiles/qt4-mac/qt-everywhere-opensource-src-4.8.6.tar.gz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 10  230M   10 23.1M    0     0   477k      0  0:08:14  0:00:49  0:07:25  569k

Again, this is reproducible for me. Can't be the disk on the Mac which is an SSD. Could it be that there is some slow or faulty harddisk on the servers there?

[agraef (Albert Graef)]

Replying to aggraef@…:

Again, this is reproducible for me. Can't be the disk on the Mac which is an SSD. Could it be that there is some slow or faulty harddisk on the servers there?

No, that's no explanation either, as the same file downloads ok on my Linux box.

[agraef (Albert Graef)]

There's something really weird going on. The file downloads in a few seconds on my MacBook if I go through ​ftp://ftp.fau.de instead of ​http://nue.de.packages.macports.org. So it seems that just the httpd connections are broken in some way. Does anyone have an idea what might be causing this?

curl -L ftp://ftp.fau.de/macports/packages/gtk3/gtk3-3.10.0_0+x11.darwin_10.x86_64.tbz2 > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.2M  100 13.2M    0     0   648k      0  0:00:20  0:00:20 --:--:--  564k

[agraef (Albert Graef)]

In any case it seems obvious that it's my MacBook and not the server which is to blame here. If anyone has an idea what I can do to fix it on my side, please let me know, but otherwise the right course of action is to close this report. I'll follow up on it if I find a resolution. Thanks.

[neverpanic (Clemens Lang)]

Un-CCing the admins in this case. There's nothing to fix for you here – if it turns out there is, I'll get back to you. Thank you for your time.

The IPv6 address you have is a local one, not a globally routed one, so you do not in fact have an IPv6 internet connection. You can try anyway using curl -6, but I assume it'll just tell you "no route to host". Similarly, you can force IPv4 using curl -4.

Both files download just fine at full speed for me. Do you happen to have some internet security/firewall/antivirus/etc. installed that might be filtering HTTP traffic? Maybe your network is forcing a transparent proxy onto you? Do you have a VPN/SSH tunnel or a similar method to try from outside your local network?

[agraef (Albert Graef)]

Replying to cal@…:

The IPv6 address you have is a local one, not a globally routed one, so you do not in fact have an IPv6 internet connection. You can try anyway using curl -6, but I assume it'll just tell you "no route to host". Similarly, you can force IPv4 using curl -4.

You're right. My DSL router connects to the ISP via an IPv4 connection, but that's the same for all machines in my home network.

Both files download just fine at full speed for me. Do you happen to have some internet security/firewall/antivirus/etc. installed that might be filtering HTTP traffic?

Nothing beyond what my DSL router does. That's just a plain Fritz!Box as you can probably find in every second home with an internet connection in Germany.

Maybe your network is forcing a transparent proxy onto you?

Not in the home network. Maybe my ISP does? Is there a way I can detect this?

But anyway, AFAICT all this should affect all the machines in my home network, and I only see these slowdowns on the Mac. So, unless there's a weird interaction between the Mac's network stack and my router, I don't see how this could have an impact.

Do you have a VPN/SSH tunnel or a similar method to try from outside your local network?

Yeah, I could try using a VPN to our university network, but I still need to set that up on the Mac. And of course I can give it another try next time I'm in the office.