wiki:Mirroring

Context Navigation

Version 8 (modified by ryandesign (Ryan Carsten Schmidt), 9 years ago) (diff)
add mime type for icns

Mirroring

Thank you for your interest in providing a public MacPorts mirror server to join our existing list of mirrors. Making MacPorts content available in more locations makes MacPorts faster.

As of March 2016 these instructions are preliminary. If you find inaccuracies or have other comments or questions, please let us know by contacting portmgr.

Contact us

First, please contact portmgr to let us know you would like to create a mirror. Please tell us the following information:

The name and web page URL of the organization providing the mirror, so that we can credit you properly.
The city, state/province and ISO 3166-1 alpha-2 country code where your mirror is located and the IATA airport code of the closest airport. The airport code and country code will become your MacPorts mirror name.
Your mirror's existing hostname. We will create new hostnames abc.xy.distfiles.macports.org, abc.xy.packages.macports.org and abc.xy.rsync.macports.org (where "abc" is your airport code and "xy" is your country code) CNAMEd to your existing hostname.
The email address of the server's administrator. We will list this on our mirrors page and MacPorts users might report problems to this address.
Your mirror's upstream Internet connection speed and whether or not your mirror supports IPv6. We will list this information on our mirrors page.

We will review your request to become a mirror and will let you know if there are any problems. Meanwhile you can follow the rest of these instructions to get your mirror ready. We reserve the right to refuse a request to become a mirror for any reason, or to remove existing mirrors from use at any time and without notice.

If in the future you wish to temporariliy or permanently discontinue providing your mirror, please let us know so that we can make the appropriate changes.

Requirements

Running a MacPorts mirror will use a possibly significant amount of bandwidth. Our primary server in California, for example, serves around 250GB of data via http and another 70GB of data via rsync per day. For http traffic, MacPorts chooses the closest mirror automatically, so how much traffic your mirror sees depends on the number of MacPorts users for whom your mirror is the closest. Ensure you have the permission of your network provider to run such a mirror, and that this amount of traffic will not be a problem for you.

You'll need a server with at least 1TB of available disk space on a case-sensitive filesystem. You'll need a web server on port 80 such as Apache httpd or nginx on which you can configure new virtual hosts, and the rsync daemon running on port 873. The server should have a hostname and a static IPv4 address. IPv6 access is welcomed but is not required. The server must respond to pings. The server does not need to be running OS X and does not need to have MacPorts installed.

Rsync modules

The MacPorts files are distributed in four rsync modules. You should usually mirror all of them.

packages contains precompiled archives of each MacPorts port for several OS X versions. This is the largest module, around 450GB. Mirroring this module will provide the greatest benefit to MacPorts users near you since it lets them automatically receive these large files more quickly.
distfiles contains the source code tarballs from which the precompiled archives were created. There are some ports that cannot legally be distributed as binaries, so those must build from these source files on the user's computer. This is the second-largest module, around 150GB. Note: this module used to be called mpdistfiles. If you are mirroring mpdistfiles, please switch to mirroring distfiles. If your local module is called mpdistfiles, please rename it to distfiles and let us know so that we can update mirror_sites.tcl.
release contains the latest released version of the MacPorts base source code which users update by running sudo port selfupdate, as well as the collection of Portfiles—the text files MacPorts reads that tell it how to install each port—which users update by running sudo port sync. Users near you can manually configure MacPorts to user your mirror for faster speed. This small module is around 500MB.
trunk contains the latest development version of the MacPorts base source code. This small module is around 20MB.

Get the files

You'll get the files from our server via rsync. You should use the latest version of rsync.

For your initial sync, you might speed up your transfer by using a nearby existing MacPorts mirror server. For subsequent scheduled syncing, use the master rsync server, rsync-origin.macports.org.

Note: If you are syncing from rsync.macports.org, distfiles.macports.org, packages.macports.org, or a specific IP address, please switch to rsync-origin.macports.org.

There is a sample script below for mirroring these modules via rsync. You don't have to use that script if you already have other arrangements for syncing, but please use the following rsync flags, for the following reasons:

`rsync` flag	Purpose
`--compress`, `-z`	File data is compressed during transfer. The server won't re-compress files that are already compressed.
`--delete-delay`	Files which were removed on the server are removed from the mirror after the transfer is finished. This option is similar to but more efficient than `--delete-after`.
`--hard-links`, `-H`	Hard links are preserved.
`--links`, `-l`	Symbolic links (symlinks) are preserved.
`--perms`, `-p`	File and directory permissions are preserved.
`--recursive`, `-r`	Directories are copied recursively.
`--timeout=600`	If no data is transferred for 10 minutes, the connection is dropped.
`--times`, `-t`	Modification times are preserved.

Stay up to date

After the initial sync is complete, keep your mirror up to date by periodically re-running the sync script. You can do this using cron, launchd, systemd, or another scheduler. A sample cron configuration is provided below.

The sample configuration syncs every hour. It is not necessary to sync that often, but MacPorts developers commit dozens of port updates each day, so the more frequently you sync, the more likely your mirror is to have the content users are requesting. Ensure your mirror syncs at least once daily.

Rsync server

MacPorts retrieves the contents of the release and trunk modules via rsync. MacPorts comes preconfigured to use the server rsync.macports.org but users can edit their configuration to use your mirror instead. In addition, other mirror administrators may wish to copy your packages and distfiles modules. So you should run an rsync daemon to republish the modules you synced.

The MacPorts rsync modules have generic names, so we suggest you put our modules in a "macports" folder. We would prefer you not rename our modules.

Note: If you are providing the release or trunk modules under nonstandard names, please edit your rsyncd configuration to provide the module under both the nonstandard name and the standard name, and edit the Mirrors page to reflect the standard name. If you are providing the packages or distfiles modules under nonstandard names, please edit your rsyncd configuration to provide the module under the standard name, edit the Mirrors page to reflect the change, and let us know so that we can update mirror_sites.tcl and/or archive_sites.tcl.

There is a sample rsyncd configuration below.

Web server

MacPorts retrieves files from the packages and distfiles modules via http. MacPorts automatically downloads these files from whichever mirror is closest. So you should run a web server to make these modules available.

The contents of the packages module should be made available directly under http://abc.xy.packages.macports.org/ and the contents of the distfiles module should be made available directly under http://abc.xy.distfiles.macports.org/. It is not necessary to provide the contents of the release or trunk modules via http.

Note: If you are currently providing the contents of the packages or distfiles modules in a subdirectory of your MacPorts mirror hostname, or any other hostname, you should reconfigure your web server to provide the contents directly under the MacPorts mirror hostname, and let us know so we can update archive_sites.tcl and/or mirror_sites.tcl. You should configure HTTP 301 redirects from the previously-used URLs to the new ones.

Directory listings should be turned on. URL spelling correction and multiviews must be turned off. There are sample web server configurations below.

Add mirror to MacPorts

Once your server is synced and ready to accept MacPorts traffic, let us know and we will edit archive_sites.tcl and mirror_sites.tcl to add the URLs of your MacPorts mirror hostnames. Also edit the Mirrors page to add information about your server, or we can add it for you.

Sample scripts and configuration files

Sample rsync update script

This sample script can be used to mirror the MacPorts rsync modules.

For your initial sync of packages and distfiles, you can possibly make your transfer faster by uncommenting the RSYNC_URL_FORMAT line for a mirror server closer to you, but note that currently not all servers mirror all modules, and not all servers follow our module naming suggestions. For subsequent scheduled syncs, use the origin server.

#!/bin/sh

set -euo pipefail

RSYNC_DIR="/var/www/html/macports"

RSYNC_MODULES="packages distfiles release trunk"

RSYNC_URL_FORMAT="rsync://rsync-origin.macports.org/%s"
#RSYNC_URL_FORMAT="rsync://cjj.kr.rsync.macports.org/macports/%s"
#RSYNC_URL_FORMAT="rsync://fco.it.rsync.macports.org/macports-%s"
#RSYNC_URL_FORMAT="rsync://jnb.za.rsync.macports.org/macports/%s"
#RSYNC_URL_FORMAT="rsync://jog.id.rsync.macports.org/macports/%s"
#RSYNC_URL_FORMAT="rsync://lil.fr.rsync.macports.org/%s"
#RSYNC_URL_FORMAT="rsync://mse.uk.rsync.macports.org/%s.macports.org"
#RSYNC_URL_FORMAT="rsync://nou.nc.rsync.macports.org/macports/%s.macports.org"
#RSYNC_URL_FORMAT="rsync://nue.de.rsync.macports.org/macports/%s"
#RSYNC_URL_FORMAT="rsync://osl.no.rsync.macports.org/%s"
#RSYNC_URL_FORMAT="rsync://sea.us.rsync.macports.org/%s"
#RSYNC_URL_FORMAT="rsync://sjc.us.rsync.macports.org/%s"

RSYNC=rsync
RSYNC_ARGS="--compress --delete-delay --hard-links --links --no-motd --perms --recursive --stats --timeout=600 --times"
if [ -t 1 ]; then
    RSYNC_ARGS="$RSYNC_ARGS --info=progress2"
fi

for RSYNC_MODULE in $RSYNC_MODULES; do
    RSYNC_URL="$(printf "$RSYNC_URL_FORMAT" "$RSYNC_MODULE")"
    echo "Syncing $RSYNC_MODULE from $RSYNC_URL"
    $RSYNC $RSYNC_ARGS "$RSYNC_URL" "$RSYNC_DIR/$RSYNC_MODULE"
    echo
done

Sample cron configuration

This sample cron configuration runs the above sync script every hour at 30 minutes past the hour:

30 * * * * /usr/local/bin/macports-rsync.sh 2>&1 >> /var/log/macports-rsync.log

So that not all mirrors are trying to connect to the origin server at the same time, please change "30" to a random number between 0 and 59.

Sample Apache httpd configuration

This sample configuration can be used to serve MacPorts files with Apache httpd.

<Directory /var/www/html/macports>
    Order allow,deny
    Allow from all
    AllowOverride None
    Options +Indexes -FollowSymLinks -MultiViews
    IndexOptions Charset=UTF-8 FancyIndexing IconsAreLinks HTMLTable NameWidth=* SuppressDescription
    AddType application/macbinhex-40 .hqx
    AddType application/octet-stream .bin .exe .rmd160
    AddType application/vnd.apple.installer+xml .dist .distz .mpkg .pkg
    AddType application/x-7z-compressed .7z
    AddType application/x-apple-diskimage .dmg .image .img .smi
    AddType application/x-bzip2 .bz2 .tbz .tbz2
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType application/x-iso9660-image .iso
    AddType application/x-lzip .lz
    AddType application/x-lzma .lzma .tlz
    AddType application/x-rar-compressed .rar
    AddType application/x-stuffit .sit
    AddType application/x-stuffitx .sitx
    AddType application/x-tar .tar
    AddType application/x-tcl .tcl
    AddType application/x-xz .txz .xz
    AddType application/zip .zip
    AddType image/icns .icns
</Directory>
<VirtualHost *:80>
    ServerName abc.xy.distfiles.macports.org
    ServerAlias distfiles.macports.org
    DocumentRoot /var/www/html/macports/distfiles
    UseCanonicalName On
    <IfModule mod_speling.c>
        CheckSpelling Off
    </IfModule>

    Header set "Cache-Control" "max-age=2592000, public"
    <LocationMatch "/$">
        Header set "Cache-Control" "max-age=300, public"
    </LocationMatch>
    <LocationMatch "^/ports\.tar\.gz$">
        Header set "Cache-Control" "max-age=1800, public"
    </LocationMatch>
</VirtualHost>
<VirtualHost *:80>
    ServerName abc.xy.packages.macports.org
    ServerAlias packages.macports.org
    DocumentRoot /var/www/html/macports/packages
    UseCanonicalName On
    <IfModule mod_speling.c>
        CheckSpelling Off
    </IfModule>

    Header set "Cache-Control" "max-age=2592000, public"
    <LocationMatch "/$">
        Header set "Cache-Control" "max-age=300, public"
    </LocationMatch>
</VirtualHost>

If you are serving MacPorts files from subdirectories on any of your hostnames, use Redirect directives to redirect those requests to their new locations. For example, these directives show how to redirect requests for http://hostname/pub/macports/packages/... to http://abc.xy.packages.macports.org/... and requests for http://hostname/pub/macports/distfiles/... to http://abc.xy.distfiles.macports.org/...:

    Redirect 301 /pub/macports/packages http://abc.xy.packages.macports.org
    Redirect 301 /pub/macports/distfiles http://abc.xy.distfiles.macports.org

Place these directives inside the VirtualHost directives for each hostname from which you were serving the files at the old URLs, or use an include file.

Sample nginx configuration

This sample configuration can be used to serve MacPorts files with nginx.

types {
    application/macbinhex-40 hqx;
    application/octet-stream bin exe rmd160;
    application/vnd.apple.installer+xml dist distz mpkg pkg;
    application/x-7z-compressed 7z;
    application/x-apple-diskimage dmg image img smi;
    application/x-bzip2 bz2 tbz tbz2;
    application/x-compress Z;
    application/x-gzip gz tgz;
    application/x-iso9660-image iso;
    application/x-lzip lz;
    application/x-lzma lzma tlz;
    application/x-rar-compressed rar;
    application/x-stuffit sit;
    application/x-stuffitx sitx;
    application/x-tar tar;
    application/x-tcl tcl;
    application/x-xz txz xz;
    application/zip zip;
    image/icns icns;
}
server {
    listen 80;
    listen [::]:80;
    server_name abc.xy.distfiles.macports.org distfiles.macports.org;
    root /var/www/html/macports/distfiles;
    server_name_in_redirect on;
    autoindex on;

    add_header Cache-Control "max-age=2592000, public";
    location ~ /$ {
        add_header Cache-Control "max-age=300, public";
    }
    location = /ports.tar.gz {
        add_header Cache-Control "max-age=1800, public";
    }
}
server {
    listen 80;
    listen [::]:80;
    server_name abc.xy.packages.macports.org packages.macports.org;
    root /var/www/html/macports/packages;
    server_name_in_redirect on;
    autoindex on;

    add_header Cache-Control "max-age=2592000, public";
    location ~ /$ {
        add_header Cache-Control "max-age=300, public";
    }
}

If you are serving MacPorts files from subdirectories on any of your hostnames, use return directives inside a location directive to redirect those requests to their new locations. For example, these directives show how to redirect requests for http://hostname/pub/macports/packages/... to http://abc.xy.packages.macports.org/... and requests for http://hostname/pub/macports/distfiles/... to http://abc.xy.distfiles.macports.org/...:

    location ^~ /pub/macports/packages(/.*)?$ {
        return 301 http://localhost.packages.macports.org$1;
    }
    location ^~ /pub/macports/distfiles(/.*)?$ {
        return 301 http://localhost.distfiles.macports.org$1;
    }

Place these directives inside the server directives for each hostname from which you were serving the files at the old URLs, or use an include file.

Sample rsyncd configuration

TODO

Download in other formats:

Plain Text