Mirroring

Thank you for your interest in providing a public MacPorts mirror server to join our existing list of mirrors. Making MacPorts content available in more locations makes MacPorts faster.

As of March 2016 these instructions are preliminary. If you find inaccuracies or have other comments or questions, please let us know by contacting ​portmgr.

Table of Contents

1. Contact us
2. Requirements
3. Rsync modules
4. Get the files
5. Stay up to date
6. Rsync server
7. Web server
8. Add mirror to MacPorts
9. Sample scripts and configuration files
   1. Sample rsync update script
   2. Sample cron configuration
   3. Sample Apache httpd configuration
   4. Sample nginx configuration
   5. Sample rsyncd configuration

Contact us

First, please contact ​portmgr to let us know you would like to create a mirror. Please tell us the following information:

• The name and web page URL of the organization providing the mirror, so that we can credit you properly.
• The city, state/province and ​ISO 3166-1 alpha-2 country code where your mirror is located and the ​IATA airport code of the closest airport. The airport code and country code will become your MacPorts mirror name.
• GPS coordinates of your mirror's approximate location that we may display on a map. You may provide the coordinates of the above airport if you don't want to be more specific.
• Your mirror's existing hostname. We will create new hostnames abc.xy.distfiles.macports.org, abc.xy.packages.macports.org and abc.xy.rsync.macports.org (where "abc" is your airport code and "xy" is your country code) CNAMEd to your existing hostname.
• The email address of the server's administrator. We will list this on our mirrors page and MacPorts users might report problems to this address.
• Your mirror's upstream Internet connection speed and whether or not your mirror supports IPv6. We will list this information on our mirrors page.

We will review your request to become a mirror and will let you know if there are any problems. Meanwhile you can follow the rest of these instructions to get your mirror ready. We reserve the right to refuse a request to become a mirror for any reason, or to remove existing mirrors from use at any time and without notice.

If in the future you wish to temporarily or permanently discontinue providing your mirror, please let us know so that we can make the appropriate changes.

Requirements

Running a MacPorts mirror will use a possibly significant amount of bandwidth. Our primary server in California, for example, serves around 250GB of data via http and another 70GB of data via rsync per day. For http traffic, MacPorts chooses the closest mirror automatically, so how much traffic your mirror sees depends on the number of MacPorts users for whom your mirror is the closest. Ensure you have the permission of your network provider to run such a mirror, and that this amount of traffic will not be a problem for you.

You'll need a server with at least 1TB of available disk space on a case-sensitive filesystem. You'll need a web server on port 80 such as Apache httpd or nginx on which you can configure new virtual hosts, and the rsync daemon running on port 873. The server should have a hostname and a static IPv4 address. IPv6 access is welcomed but is not required. The server must respond to pings. The server does not need to be running OS X and does not need to have MacPorts installed.

Rsync modules

The MacPorts files are distributed in four rsync modules. You should usually mirror all of them.

• packages contains precompiled archives of each MacPorts port for several OS X versions. This is the largest module, around 450GB. Mirroring this module will provide the greatest benefit to MacPorts users near you since it lets them automatically receive these large files more quickly.
• distfiles contains the source code tarballs from which the precompiled archives were created. There are some ports that cannot legally be distributed as binaries, so those must build from these source files on the user's computer. This is the second-largest module, around 150GB. Note: this module used to be called mpdistfiles. If you are mirroring mpdistfiles, please switch to mirroring distfiles. If your local module is called mpdistfiles, please rename it to distfiles and let us know so that we can update mirror_sites.tcl.
• release contains the latest released version of the MacPorts base source code which users update by running sudo port selfupdate, as well as the collection of Portfiles—the text files MacPorts reads that tell it how to install each port—which users update by running sudo port sync. Users near you can manually configure MacPorts to user your mirror for faster speed. This small module is around 500MB.
• trunk contains the latest development version of the MacPorts base source code. This small module is around 20MB.

We may in the future offer a single macports rsync module containing the above four modules as subdirectories, which is the configuration we recommend for mirrors.

Get the files

You'll get the files from our server via ​rsync. You should use the latest version of rsync.

For your initial sync, you might speed up your transfer by using a nearby existing MacPorts mirror server. For subsequent scheduled syncing, use the master rsync server, rsync-origin.macports.org.

Note: If you are syncing from a different hostname such as rsync.macports.org, or from a specific IP address, please switch to rsync-origin.macports.org.

There is a sample script below for mirroring these modules via rsync. You don't have to use that script if you already have other arrangements for syncing, but please use the following rsync flags, for the following reasons:

rsync flag	Purpose
--compress, -z	File data is compressed during transfer. The server won't re-compress files that are already compressed.
--delete-delay	Files which were removed on the server are removed from the mirror after the transfer is finished. This option is similar to but more efficient than --delete-after.
--hard-links, -H	Hard links are preserved.
--links, -l	Symbolic links (symlinks) are preserved.
--perms, -p	File and directory permissions are preserved.
--recursive, -r	Directories are copied recursively.
--timeout=600	If no data is transferred for 10 minutes, the connection is dropped.
--times, -t	Modification times are preserved.

Stay up to date

After the initial sync is complete, keep your mirror up to date by periodically re-running the sync script. You can do this using cron, launchd, systemd, or another scheduler. A sample cron configuration is provided below.

The sample configuration syncs every hour. It is not necessary to sync that often, but MacPorts developers commit dozens of port updates each day, so the more frequently you sync, the more likely your mirror is to have the content users are requesting. Ensure your mirror syncs at least once daily.

Rsync server

MacPorts retrieves files from release and trunk via rsync. MacPorts comes preconfigured to use the server rsync.macports.org but users can edit their configuration to use your mirror instead. In addition, other mirror administrators may wish to copy your packages and distfiles. So you should run an rsync daemon to republish the files you synced.

The MacPorts rsync modules have generic names, so we suggest you put our files in subdirectories of a single "macports" module. We would prefer you not rename those subdirectories. If your site convention is to use the original server's hostname as the rsync module name, please use the name rsync.macports.org (even though you are actually syncing from rsync-origin.macports.org).

Note: If you are providing the modules under nonstandard names, please edit your rsyncd configuration to provide the files under a single module with subdirectories, but continue to serve the files from your old module names so that existing users can still find the files. If you are already providing a single module but with nonstandard subdirectory names, rename the subdirectories to their standard names but add symlinks at their previous names. Edit the Mirrors page to reflect your changes, and if you changed your configuration for packages or distfiles let us know so that we can update mirror_sites.tcl and/or archive_sites.tcl.

There is a sample rsyncd configuration below.

Web server

MacPorts retrieves files from packages and distfiles via http. MacPorts automatically downloads these files from whichever mirror is closest. So you should run a web server to make these modules available.

The contents of the packages module should be made available directly under http://abc.xy.packages.macports.org/ and the contents of the distfiles module should be made available directly under http://abc.xy.distfiles.macports.org/. It is not necessary to provide the contents of the release or trunk modules via http.

Note: If you are currently providing the contents of the packages or distfiles modules in a subdirectory of your MacPorts mirror hostname, you should reconfigure your web server to provide the contents directly under the MacPorts mirror hostname, and let us know so we can update archive_sites.tcl and/or mirror_sites.tcl. You should configure HTTP 301 redirects from the previously-used URLs to the new ones.

Directory listings should be turned on. URL spelling correction and multiviews must be turned off. There are sample web server configurations below.

Add mirror to MacPorts

Once your server is synced and ready to accept MacPorts traffic, let us know and we will edit archive_sites.tcl and mirror_sites.tcl to add the URLs of your MacPorts mirror hostnames. Also edit the Mirrors page to add information about your server, or we can add it for you.

Sample scripts and configuration files

Sample rsync update script

This sample script can be used to mirror the MacPorts rsync modules.

For your initial sync of packages and distfiles, you can possibly make your transfer faster by uncommenting the RSYNC_URL_FORMAT line for a mirror server closer to you, but note that currently not all servers mirror all modules, and not all servers follow our module naming suggestions. For subsequent scheduled syncs, use the origin server.

#!/bin/bash

set -euo pipefail

RSYNC_DIR="/var/www/html/macports"

RSYNC_MODULES="packages distfiles release trunk"

RSYNC_URL_FORMAT="rsync://rsync-origin.macports.org/%s/"
#RSYNC_URL_FORMAT="rsync://cjj.kr.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://fco.it.rsync.macports.org/macports-%s/"
#RSYNC_URL_FORMAT="rsync://jnb.za.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://jog.id.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://lil.fr.rsync.macports.org/%s/"
#RSYNC_URL_FORMAT="rsync://mse.uk.rsync.macports.org/%s.macports.org/"
#RSYNC_URL_FORMAT="rsync://nou.nc.rsync.macports.org/macports/%s.macports.org/"
#RSYNC_URL_FORMAT="rsync://nue.de.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://osl.no.rsync.macports.org/%s/"
#RSYNC_URL_FORMAT="rsync://sea.us.rsync.macports.org/%s/"
#RSYNC_URL_FORMAT="rsync://sjc.us.rsync.macports.org/%s/"

RSYNC=rsync
RSYNC_ARGS="--compress --delete-delay --hard-links --links --no-motd --perms --recursive --stats --timeout=600 --times"
if [ -t 1 ]; then
    RSYNC_ARGS="$RSYNC_ARGS --info=progress2"
fi

for RSYNC_MODULE in $RSYNC_MODULES; do
    RSYNC_URL="$(printf "$RSYNC_URL_FORMAT" "$RSYNC_MODULE")"
    echo "Syncing $RSYNC_MODULE from $RSYNC_URL"
    $RSYNC $RSYNC_ARGS "$RSYNC_URL" "$RSYNC_DIR/$RSYNC_MODULE"
    echo
done

Sample cron configuration

This sample cron configuration runs the above sync script every hour at 30 minutes past the hour:

30 * * * * /usr/local/bin/macports-rsync.sh 2>&1 >> /var/log/macports-rsync.log

So that not all mirrors are trying to connect to the origin server at the same time, please change "30" to a random number between 0 and 59.

Sample Apache httpd configuration

This sample configuration can be used to serve MacPorts files with Apache httpd.

<Directory /var/www/html/macports>
    Order allow,deny
    Allow from all
    AllowOverride None
    Options +Indexes -FollowSymLinks -MultiViews
    IndexOptions Charset=UTF-8 FancyIndexing IconsAreLinks HTMLTable NameWidth=* SuppressDescription
    AddType application/macbinhex-40 .hqx
    AddType application/octet-stream .bin .exe .rmd160
    AddType application/vnd.apple.installer+xml .dist .distz .mpkg .pkg
    AddType application/x-7z-compressed .7z
    AddType application/x-apple-diskimage .dmg .image .img .smi
    AddType application/x-bzip2 .bz2 .tbz .tbz2
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType application/x-iso9660-image .iso
    AddType application/x-lzip .lz
    AddType application/x-lzma .lzma .tlz
    AddType application/x-rar-compressed .rar
    AddType application/x-stuffit .sit
    AddType application/x-stuffitx .sitx
    AddType application/x-tar .tar
    AddType application/x-tcl .tcl
    AddType application/x-xz .txz .xz
    AddType application/zip .zip
    AddType image/icns .icns
</Directory>
<VirtualHost *:80>
    ServerName abc.xy.distfiles.macports.org
    ServerAlias distfiles.macports.org
    DocumentRoot /var/www/html/macports/distfiles
    UseCanonicalName On
    <IfModule mod_speling.c>
        CheckSpelling Off
    </IfModule>

    Header set "Cache-Control" "max-age=2592000, public"
    <LocationMatch "/$">
        Header set "Cache-Control" "max-age=300, public"
    </LocationMatch>
    <LocationMatch "^/ports\.tar\.gz$">
        Header set "Cache-Control" "max-age=1800, public"
    </LocationMatch>
</VirtualHost>
<VirtualHost *:80>
    ServerName abc.xy.packages.macports.org
    ServerAlias packages.macports.org
    DocumentRoot /var/www/html/macports/packages
    UseCanonicalName On
    <IfModule mod_speling.c>
        CheckSpelling Off
    </IfModule>

    Header set "Cache-Control" "max-age=2592000, public"
    <LocationMatch "/$">
        Header set "Cache-Control" "max-age=300, public"
    </LocationMatch>
</VirtualHost>

If you are serving MacPorts files from subdirectories of your MacPorts mirror hostnames, use Redirect directives to redirect those requests to their new locations. For example, these directives show how to redirect requests for http://hostname/pub/macports/packages/... to http://abc.xy.packages.macports.org/... and requests for http://hostname/pub/macports/distfiles/... to http://abc.xy.distfiles.macports.org/...:

    Redirect 301 /pub/macports/packages http://abc.xy.packages.macports.org
    Redirect 301 /pub/macports/distfiles http://abc.xy.distfiles.macports.org

Place these directives inside the VirtualHost directives for your MacPorts mirror hostnames.

Sample nginx configuration

This sample configuration can be used to serve MacPorts files with nginx.

types {
    application/macbinhex-40 hqx;
    application/octet-stream bin exe rmd160;
    application/vnd.apple.installer+xml dist distz mpkg pkg;
    application/x-7z-compressed 7z;
    application/x-apple-diskimage dmg image img smi;
    application/x-bzip2 bz2 tbz tbz2;
    application/x-compress Z;
    application/x-gzip gz tgz;
    application/x-iso9660-image iso;
    application/x-lzip lz;
    application/x-lzma lzma tlz;
    application/x-rar-compressed rar;
    application/x-stuffit sit;
    application/x-stuffitx sitx;
    application/x-tar tar;
    application/x-tcl tcl;
    application/x-xz txz xz;
    application/zip zip;
    image/icns icns;
}
server {
    listen 80;
    listen [::]:80;
    server_name abc.xy.distfiles.macports.org distfiles.macports.org;
    root /var/www/html/macports/distfiles;
    server_name_in_redirect on;
    autoindex on;

    add_header Cache-Control "max-age=2592000, public";
    location ~ /$ {
        add_header Cache-Control "max-age=300, public";
    }
    location = /ports.tar.gz {
        add_header Cache-Control "max-age=1800, public";
    }
}
server {
    listen 80;
    listen [::]:80;
    server_name abc.xy.packages.macports.org packages.macports.org;
    root /var/www/html/macports/packages;
    server_name_in_redirect on;
    autoindex on;

    add_header Cache-Control "max-age=2592000, public";
    location ~ /$ {
        add_header Cache-Control "max-age=300, public";
    }
}

If you are serving MacPorts files from subdirectories of your MacPorts mirror hostnames, use return directives inside a location directive to redirect those requests to their new locations. For example, these directives show how to redirect requests for http://hostname/pub/macports/packages/... to http://abc.xy.packages.macports.org/... and requests for http://hostname/pub/macports/distfiles/... to http://abc.xy.distfiles.macports.org/...:

    location ^~ /pub/macports/packages(/.*)?$ {
        return 301 http://abc.xy.packages.macports.org$1;
    }
    location ^~ /pub/macports/distfiles(/.*)?$ {
        return 301 http://abc.xy.distfiles.macports.org$1;
    }

Place these directives inside the server directives for your MacPorts mirror hostnames.

Sample rsyncd configuration

This sample rsyncd.conf file can be used to serve MacPorts files with the rsync daemon.

dont compress = 7z ace avi bin bz2 deb distz dmg gem gif gpg gz \
    icns image img iso jar jpeg jpg lz lzma lzo mov mp3 mp4 mpkg msi \
    ogg pdf pkg png rar rmd160 rpm rzip sig sit sitx smi tbz tbz2 \
    tgz tlz ttf txz xz Z zip
forward lookup = false
gid = nobody
list = true
log file = /var/log/rsyncd/rsyncd.log
max connections = 0
pid file = /var/run/rsyncd.pid
read only = true
reverse lookup = false
timeout = 600
transfer logging = true
uid = nobody
use chroot = no

[macports]
    comment = MacPorts release, trunk, distfiles and packages
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports

Start rsync with:

rsync --daemon --config=/etc/rsyncd.conf

If you are serving individual rsync modules, move those directories into a single new macports module but continue to serve the old modules under your existing names so that users using the old names can still get the files. For example:

cd /var/www/html
mkdir macports
mv packages distfiles release trunk macports/

Adjust your rsyncd.conf to the new directory locations:

[release]
    comment = MacPorts release (deprecated; use "macports/release" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/release

[trunk]
    comment = MacPorts trunk (deprecated; use "macports/trunk" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/trunk

[distfiles]
    comment = MacPorts distfiles (deprecated; use "macports/distfiles" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/distfiles

[mpdistfiles]
    comment = MacPorts distfiles (deprecated; use "macports/distfiles" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/distfiles

[packages]
    comment = MacPorts packages (deprecated; use "macports/packages" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/packages

If you are already serving a single module but with subdirectories with nonstandard names, move the subdirectories to their standard names and add a symlink so that users using the old nonstandard names can still get the files. For example, if you are currently serving mpdistfiles, rename it to distfiles and add an mpdistfiles symlink:

cd /var/www/html/macports
mv mpdistfiles distfiles
ln -s distfiles mpdistfiles