rsync.macports.org is very slow

[lpancescu (Laurențiu Păncescu)]

I installed MacPorts 2.3.4 using the official pkg installer on OS X 10.11.1 (first on Saturday, and now again, to try to figure out why it takes 52 minutes on my Mac Mini "late 2009"). The last 30 minutes of that time are spent by tclsh building the PortIndex, already reported as bug #49050. The first 20 minutes seem to be just the rsync-ing of base.tar and ports.tar, with a transfer rate jumping between 14KB/s and 33KB/s, as seen in Activity Monitor.

To make sure this isn't due to a regression in El Capitan, I tried rsync-ing base.tar manually from my Fedora netbook, and compared the main server, rsync.macports.org, with the French mirror (I'm in Germany):

$ time rsync -rztv rsync://rsync.macports.org/release/tarballs/base.tar .
receiving incremental file list
base.tar

sent 48 bytes  received 14,253,729 bytes  21,678.75 bytes/sec
total size is 27,013,120  speedup is 1.90

real	10m57.161s
user	0m0.696s
sys	0m0.907s
$ rm base.tar 
$ time rsync -rztv rsync://lil.fr.rsync.macports.org/release/tarballs/base.tar .
 _          _        _                                                                                                                                               
| |   o    | |      | |                                                                                                                                              
| |        | |      | |    ,_         ,_      ,             _  _      __        _  _  _      __,     __       _     __     ,_     _|_    ,        __     ,_      __, 
|/    |    |/       |/    /  |       /  |    / \_  |   |   / |/ |    /         / |/ |/ |    /  |    /       |/ \_  /  \_  /  |     |    / \_     /  \_  /  |    /  | 
|__/  |_/  |__/  o  |__/     |_/  o     |_/   \/    \_/|/    |  |_/  \___/  o    |  |  |_/  \_/|_/  \___/   |__/   \__/      |_/   |_/   \/   o  \__/      |_/  \_/|/
                    |\                                /|                                                   /|                                                     /| 
                    |/                                \|                                                   \|                                                     \| 


    rsync://lil.fr.rsync.macports.org
    rsync://rsync.ionic.de
    rsync://macports.rsync.ionic.de

This is your local MacPorts mirror.

You may sync whenever and how often you want,
  but keep in mind, that we enforce a maximum of 5 (five)
  concurrent connections per host. If you feel this is too little,
  please contact the administration.

The server is physically located in Roubaix, France
  and hosted by OVH. It though features a German RIPE failover IP.

If any problems or questions arise, please contact
  Mihai Moldovan <ionic@ionic.de>, Karlsruhe, Germany (primary) or
  Joerg Hoffmann <jh2000@root24.eu>, Nordholz, Germany.


receiving incremental file list
base.tar

sent 43 bytes  received 14,254,789 bytes  1,357,603.05 bytes/sec
total size is 27,013,120  speedup is 1.90

real	0m10.873s
user	0m0.719s
sys	0m0.782s

Is rsync.macports.org intentionally limiting its bandwidth for clients outside the USA? Unfortunately, I didn't see any option of selecting a different mirror in the pkg installer.

[ryandesign (Ryan Carsten Schmidt)]

Replying to lpancescu@…:

Is rsync.macports.org intentionally limiting its bandwidth for clients outside the USA?

We have never been informed of the existence of such a limit.

Is fetching distfiles from distfiles.macports.org slow as well? That's the same server.

Unfortunately, I didn't see any option of selecting a different mirror in the pkg installer.

Correct, there is no such option. I can't recall anyone ever having requested such an option be added before.

[lpancescu (Laurențiu Păncescu)]

Replying to ryandesign@…:

Is fetching distfiles from distfiles.macports.org slow as well? That's the same server.

I used curl to see how fast I can get the directory index, which is big enough to measure:

$ curl --compressed http://distfiles.macports.org > /dev/null  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1814k    0 1814k    0     0  22508      0 --:--:--  0:01:22 --:--:-- 32004

The same without compression, in case the server is under high CPU load:

$ curl http://distfiles.macports.org > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1814k    0 1814k    0     0  20980      0 --:--:--  0:01:28 --:--:-- 42839

I also performed the Vultr speed test, downloading 100MB from their Silicon Valley data center (I guess San Jose is not that far from San Francisco), to see if there are general network issues:

$ curl --compressed http://sjo-ca-us-ping.vultr.com/vultr.com.100MB.bin -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  100M  100  100M    0     0  1499k      0  0:01:08  0:01:08 --:--:-- 1588k

Download the directory index from the Korean mirror, since Seattle seems down:

$ curl --compressed http://cjj.kr.distfiles.macports.org > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1357k  100 1357k    0     0   311k      0  0:00:04  0:00:04 --:--:--  311k

Update: Seattle now works again - here are the results:

$ curl --compressed http://sea.us.distfiles.macports.org > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1722k  100 1722k    0     0   452k      0 --:--:--  0:00:03 --:--:--  452k

I'm using MacPorts since the beginning of 2010, and rsync.macports.org was always very fast for me; I first noticed "port selfupdate" taking longer and longer sometime last year, so I switched to the French mirror without looking into what actually happens (until last Saturday).

Would it be difficult to allow the user to select a mirror in the installer, in case rsync.macports.org cannot be fixed?

[lpancescu (Laurențiu Păncescu)]

I just realized, packet loss could lead to a low transfer rate or even to a stalled TCP connection (due to the TCP sliding window protocol). Let's test that theory:

• French mirror

  $ ping -q -c 100 lil.fr.rsync.macports.org
  PING lil.fr.rsync.macports.org (87.98.244.45): 56 data bytes
  
  --- lil.fr.rsync.macports.org ping statistics ---
  100 packets transmitted, 100 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 35.509/36.627/41.092/0.756 ms
  

• Canadian mirror

  $ ping -q -c 100 ykf.ca.rsync.macports.org
  PING mirror.csclub.uwaterloo.ca (129.97.134.71): 56 data bytes
  
  --- mirror.csclub.uwaterloo.ca ping statistics ---
  100 packets transmitted, 100 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 123.321/125.046/134.289/1.397 ms
  

• Korean mirror

  $ ping -q -c 100 cjj.kr.rsync.macports.org
  PING ftp.kaist.ac.kr (103.22.220.133): 56 data bytes
  
  --- ftp.kaist.ac.kr ping statistics ---
  100 packets transmitted, 100 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 308.116/546.142/815.002/149.790 ms
  

• Seattle, USA

  $ ping -q -c 100 sea.us.rsync.macports.org
  PING mirror.pnl.gov (192.101.102.2): 56 data bytes
  
  --- mirror.pnl.gov ping statistics ---
  100 packets transmitted, 100 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 188.007/189.587/190.805/0.392 ms
  

• Main server - San Francisco, USA

  $ ping -q -c 100 rsync.macports.org
  PING rsync.macports.org (17.251.224.217): 56 data bytes
  
  --- rsync.macports.org ping statistics ---
  100 packets transmitted, 94 packets received, 6.0% packet loss
  round-trip min/avg/max/stddev = 181.538/184.057/261.316/8.502 ms
  

The main server is the only one with a relatively high packet loss. Test with 1000 packages, for better accuracy:

$ ping -q -c 1000 rsync.macports.org
PING rsync.macports.org (17.251.224.217): 56 data bytes

--- rsync.macports.org ping statistics ---
1000 packets transmitted, 958 packets received, 4.2% packet loss
round-trip min/avg/max/stddev = 181.073/182.229/190.005/0.674 ms

[danielluke (Daniel J. Luke)]

A traceroute would probably be helpful. For what it's worth, I can replicate packet loss to rsync.macports.org from within the US (in my case, it looks like the packet loss is happening at the interconnection between cogent and level3).

[danielluke (Daniel J. Luke)]

Replying to dluke@…:

A traceroute would probably be helpful. For what it's worth, I can replicate packet loss to rsync.macports.org from within the US (in my case, it looks like the packet loss is happening at the interconnection between cogent and level3).

and from another host (on a different network) I also see packet loss at the border of the level3 network - so I suspect it's a level3 issue.

[lpancescu (Laurențiu Păncescu)]

Tracerouting from here only shows Level3 from Frankfurt (Germany) to Apple:

$ traceroute rsync.macports.org
traceroute to rsync.macports.org (17.251.224.217), 64 hops max, 52 byte packets
 1  speedport.ip (192.168.2.1)  3.285 ms  1.758 ms  1.827 ms
 2  217.0.117.176 (217.0.117.176)  20.358 ms  20.004 ms  19.674 ms
 3  87.186.197.34 (87.186.197.34)  24.682 ms  22.403 ms  25.527 ms
 4  217.239.52.114 (217.239.52.114)  23.581 ms  23.271 ms  26.549 ms
 5  te-3-0-0.edge5.frankfurt1.level3.net (4.68.62.141)  23.380 ms  29.905 ms  23.628 ms
 6  ae-3-80.edge1.sanjose2.level3.net (4.69.152.135)  241.272 ms
    ae-4-90.edge1.sanjose2.level3.net (4.69.152.199)  207.614 ms
    ae-3-80.edge1.sanjose2.level3.net (4.69.152.135)  216.037 ms
 7  ae-1-60.edge1.sanjose2.level3.net (4.69.152.7)  205.691 ms
    ae-4-90.edge1.sanjose2.level3.net (4.69.152.199)  214.055 ms  208.680 ms
 8  apple-compu.edge1.sanjose2.level3.net (4.28.172.82)  204.900 ms  208.787 ms  213.162 ms

I only tested for packet loss from hop 5 onwards and got packet loss from ae-3-80, but none (0.0%) from te-3-0-0, ae-4-90 and ae-1-60:

$ ping -q -i 0.25 -c 100 ae-3-80.edge1.sanjose2.level3.net
PING ae-3-80.edge1.sanjose2.level3.net (4.69.152.135): 56 data bytes

--- ae-3-80.edge1.sanjose2.level3.net ping statistics ---
100 packets transmitted, 97 packets received, 3.0% packet loss

[lpancescu (Laurențiu Păncescu)]

The network problem appeared to be solved today at around 9 CET, I got no packet loss from rsync.macports.org and a clean installation ended in 31 minutes, a significant improvement from the 52 minutes. However, I started to get 0.5% packet loss at around 17 CET, which kept increasing in the following hours, reaching 4.2%, like the day before. Now it's almost midnight here, and I see the packet loss dropping again to 0.5% (at this error rate, rsync manages to download at 300KB/s, which is still better than 20KB/s). Central European time is something like 8-9 hours ahead of the West Coast, if I remember correctly - perhaps the packet loss is correlated with the start of the working day in the USA. I also used ping yesterday evening from a friend though another German ISP: while I saw 6% packet loss just minutes earlier via level3.net, at her place it was only 2%, via xo.net (level3.net didn't appear anywhere in the traceroute). This probably also explains why nobody I know here managed to update iOS in the evening - "request timeout". I also see 0.1% packet loss from the Seattle mirror, which doesn't slow things down too much.

If packet loss is normal for the main server, I can only see two solutions:

1. Keep using the main server and avoid selfupdate outside of the 9-16 CET time interval
2. Switch to a local mirror, in which case it would be nice to have the option to choose one in the installer (the missing PortIndex makes mirror switching painful, see my comment on bug #49050)

[ryandesign (Ryan Carsten Schmidt)]

Since this is a level3 problem, there's not much we can do except hope level3 resolves the problem. I don't know how to contact level3 to report the problem to them. If you do, please do.

I've filed a new ticket for your idea to add a mirror choice to the installer: #49496

I didn't see a comment from you in #49050.

[danielluke (Daniel J. Luke)]

Replying to ryandesign@…:

Since this is a level3 problem, there's not much we can do except hope level3 resolves the problem. I don't know how to contact level3 to report the problem to them. If you do, please do.

realistically, only a level3 customer is going to get action on it, so the customer (Apple) should report it.

[lpancescu (Laurențiu Păncescu)]

Replying to ryandesign@…:

Since this is a level3 problem, there's not much we can do except hope level3 resolves the problem. I don't know how to contact level3 to report the problem to them. If you do, please do.

I only saw a contact form for their customers (which seems intended for bigger companies, not private people experiencing problems).

I didn't see a comment from you in #49050.

Sorry, I decided to file a new bug #49498, before submitting the comment on #49050, where I wanted to refer it by number.

[danielluke (Daniel J. Luke)]

note that today the two hosts I tested from before route to rsync.macports.com through XO instead of level3 and I'm not seeing packet loss anymore.

[keith_dart@…]

I don't see any packet loss right now.

    aplus # ping -c 100 -f  rsync.macports.org
    PING rsync.macports.org (17.251.224.217) 56(84) bytes of data.

    --- rsync.macports.org ping statistics ---
    100 packets transmitted, 100 received, 0% packet loss, time 1432ms
    rtt min/avg/max/mdev = 18.544/18.708/18.970/0.162 ms, pipe 2, ipg/ewma 14.468/18.704 ms

Performance looks pretty good, at least at the IP layer.

[lpancescu (Laurențiu Păncescu)]

In the last four days, I've been seeing the same pattern from here (Germany): no packet loss during the day, but it starts to climb around 17-18:00 CET, peaking at 4-6%, and then decreasing again (this is via Level3; via XO I saw 0% packet loss the entire time, except on Thursday evening, when it was 3% for some time). I'm not sure what the MacPorts project can do about this, it's probably normal, depending on the provider and time of day. I had always thought that companies like Google use CDNs spread all over the world to fight HTTP latency, not data transfer for larger downloads.

MacPorts already has a number of mirrors, and they work pretty well - I think redirecting users to a geographically close mirror, be it automatically or through manual configuration, is the right solution. The only problem is the initial installation: it currently takes a long time if you happen to install at the wrong moment, since the main server in always used. Fortunately, clean installs aren't something you need to do that often (probably once a year, because of a new OS X version).

[ryandesign (Ryan Carsten Schmidt)]

The primary rsync mirror is no longer hosted at macOS forge so hopefully any network issues that you had in talking with the macOS forge network no longer apply.