files hosted on ftp sites aren't getting mirrored

[jchauvel@…]

I tried to install ffmpeg-devel +gpl2+libdc1394+librtmp+nonfree+x11 on Yosemite Developer Preview 8 after compiling macports from source.

[jeremyhu (Jeremy Huddleston Sequoia)]

--->  Attempting to fetch libbluray-0.6.0.tar.bz2 from ftp://ftp.videolan.org/pub/videolan/libbluray/0.6.0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  630k  100  630k    0     0  61542      0  0:00:10  0:00:10 --:--:--  125k

It downloads fine for me. I suspect the issue is with seeding our mirrors with content that is originally in an ftp server.

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #45267, about grace.

Shree: Can we get the MacPorts distfiles mirrors capable of fetching from FTP sites again, please? This used to work.

And/or: Daniel: if we're still using your proxy, can you check if it's running and working? Thanks.

[danielluke (Daniel J. Luke)]

Something at least is still configured to use the proxy (which is running). Most recent log:

1412623895.366   3459 17.251.224.231 TCP_MISS/404 4404 GET ftp://ftp.kde.org/pub/kde/stable/kdevelop/4.6.0/src/patch-plugins_appwizard_appwizardplugin.cpp.diff - HIER_DIRECT/2001:4ca0:100::10:180 text/html

That corresponds to Mon, 06 Oct 2014 19:31:35 GMT.

I do see hits for libbluray 0.6.0

1406447710.686  11936 17.251.224.232 TCP_MISS/200 645738 GET ftp://ftp.videolan.org/pub/videolan/libbluray/0.6.0/libbluray-0.6.0.tar.bz2 - HIER_NONE/- application/octet-stream
1406447710.686  14743 17.251.224.229 TCP_MISS/200 645738 GET ftp://ftp.videolan.org/pub/videolan/libbluray/0.6.0/libbluray-0.6.0.tar.bz2 - HIER_DIRECT/88.191.250.2 application/octet-stream
1406447752.662  19817 17.251.224.231 TCP_MISS/200 645738 GET ftp://ftp.videolan.org/pub/videolan/libbluray/0.6.0/libbluray-0.6.0.tar.bz2 - HIER_DIRECT/88.191.250.2 application/octet-stream
1406447792.601  20244 17.251.224.230 TCP_MISS/200 645738 GET ftp://ftp.videolan.org/pub/videolan/libbluray/0.6.0/libbluray-0.6.0.tar.bz2 - HIER_DIRECT/88.191.250.2 application/octet-stream

[jmroot (Joshua Root)]

Replying to dluke@…:

Something at least is still configured to use the proxy (which is running).

That may be the buildslaves. Need to check that the machine that runs the mirror script has proxy_ftp set correctly in macports.conf, and also that its IP is in the list allowed by the proxy.

[danielluke (Daniel J. Luke)]

ok, let me know if it needs adjusting - Per bill (a long time ago) the proxy just allows access from 17.251.224.224/28

[ryandesign (Ryan Carsten Schmidt)]

That should still be correct:

$ dig +short ten{six,seven,eight,nine,ten}-slave.macports.org
17.251.224.229
17.251.224.230
17.251.224.231
17.251.224.232
17.251.224.234

[danielluke (Daniel J. Luke)]

What about their IPv6 addresses?

[ryandesign (Ryan Carsten Schmidt)]

I don't know how to determine that.

[danielluke (Daniel J. Luke)]

we probably need to ask the server admin ;-)

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #49078.

Keith: can you please investigate why ftp-hosted files aren't getting mirrored?

[ryandesign (Ryan Carsten Schmidt)]

Replying to dluke@…:

ok, let me know if it needs adjusting - Per bill (a long time ago) the proxy just allows access from 17.251.224.224/28

Daniel, the 17.251.224.224/28 subnet only covers the internal MacPorts servers (such as the buildbot builders). The distfiles server is where the mirroring of the files for public consumption occurs, and that's an external server, which is in the 17.251.224.208/28 subnet. Could you allow access from that subnet as well? Alternately you could allow access from the entire Mac OS Forge subnet, which is 17.251.224.0/24.

[danielluke (Daniel J. Luke)]

I've updated the squid conf and the local firewall to allow connections from 17.251.224.0/24

What is the IPv6 range that should be allowed?

[ryandesign (Ryan Carsten Schmidt)]

Replying to dluke@…:

I've updated the squid conf and the local firewall to allow connections from 17.251.224.0/24

Thanks, I think this is working. I was able to get a file via FTP using your proxy while logged into the distfiles server, which I couldn't before. We'll hopefully see later on when the automatic mirror process runs. I'll be watching for whether mesa 11.0.6 ​gets mirrored.

What is the IPv6 range that should be allowed?

I don't know much about IPv6. How can I find this information? The servers whose configurations I know how to examine (the Mac servers) have their IPv6 addresses obtained automatically (whereas their IPv4 addresses are entered manually). I can run ifconfig and see the inet6 address of all the servers, but I don't know if they're externally accessible. For example, I found the purported IPv6 address of the distfiles server, but entering it into Safari produces and error message. According to an IPv6 readiness checker, our web sites are not IPv6 ready, first of all because we don't have an AAAA record. So I'm not sure to what extent, if any, IPv6 connections are occurring within the Mac OS Forge infrastructure.

[danielluke (Daniel J. Luke)]

Replying to ryandesign@…:

I don't know much about IPv6.

There's a bunch of info online if you want to learn - I've heard good things about ​https://ipv6.he.net/certification/ (although I haven't gone through any of it).

For our purposes here, there is a block of addresses for MacOS Forge. If everything is on the same broadcast domain, it's probably a /64 (IPv6 addresses are 128 bits long and almost always a /64).

How can I find this information?

One way, would be to look at the ifconfig output on the machine(s) / VMs.

For example, the host that I run squid on has ifconfig that looks something like this:

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
	ether 10:dd:b1:a9:66:f9 
	inet6 fe80::12dd:b1ff:fea9:66f9%en0 prefixlen 64 scopeid 0x4 
	inet6 2001:418:1401:62::3 prefixlen 64 
	inet 129.250.34.3 netmask 0xfffffff8 broadcast 129.250.34.7
	nd6 options=1<PERFORMNUD>
	media: autoselect (1000baseT <full-duplex,flow-control>)
	status: active

So you can see I have an IPv4 /29 and and IPv6 /64 that the host is numbered out of (along with the link-local IPv6 address that we can ignore for now).

The servers whose configurations I know how to examine (the Mac servers) have their IPv6 addresses obtained automatically (whereas their IPv4 addresses are entered manually). I can run ifconfig and see the inet6 address of all the servers, but I don't know if they're externally accessible.

From a host that would be connecting to the proxy do ping6 geeklair.net and see if you get a reply.

or, if the stupid firewall policy blocks ICMP, you could to telnet -6 geeklair.net 80 and see if you can make a tcp connection.

If either (or both) of those succeed, we'll need to figure out which IPv6 block or blocks to add access to - or those hosts may try to connect via IPv6 and will be blocked by my local firewall (and denied by my squid configuration).

For example, I found the purported IPv6 address of the distfiles server, but entering it into Safari produces and error message. According to an IPv6 readiness checker, our web sites are not IPv6 ready, first of all because we don't have an AAAA record. So I'm not sure to what extent, if any, IPv6 connections are occurring within the Mac OS Forge infrastructure.

That's probably also something you should get working (AAAA records in DNS, website and any other services available over IPv6), but is not directly relevant to whether the machines are going to try to make outbound IPv6 connections or not.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign@…:

I'll be watching for whether mesa 11.0.6 ​gets mirrored.

It did get mirrored.

[ryandesign (Ryan Carsten Schmidt)]

Replying to dluke@…:

One way, would be to look at the ifconfig output on the machine(s) / VMs.

I did look at the ifconfig output of all the machines that would be connecting to your proxy, and they each only have a single ipv6 address starting with fe80::.

[danielluke (Daniel J. Luke)]

Replying to ryandesign@…:

I did look at the ifconfig output of all the machines that would be connecting to your proxy, and they each only have a single ipv6 address starting with fe80::.

fe80::/64 is IPv6 link-local (see RFC 4291)

Let me know whenever you get IPv6 set up and I'll adjust the firewall and squid config on my side.

[jeremyhu (Jeremy Huddleston Sequoia)]

Replying to dluke@…:

Replying to ryandesign@…:

I don't know much about IPv6.

There's a bunch of info online if you want to learn - I've heard good things about ​https://ipv6.he.net/certification/ (although I haven't gone through any of it).

You should. It's quite a good resource, and at the end, you get a nifty T-Shirt ;)

For our purposes here, there is a block of addresses for MacOS Forge. If everything is on the same broadcast domain, it's probably a /64 (IPv6 addresses are 128 bits long and almost always a /64).

I think the MacOSForge servers aren't configured for IPv6 access. xquartz and xquartz-dl only have link local IPv6 addresses, so I suspect the others are in the same boat.

[ryandesign (Ryan Carsten Schmidt)]

I'm closing this because mirroring of files hosted on FTP is working again. IPv6 discussions can happen separately.