= Mirroring

Thank you for your interest in providing a public MacPorts mirror server to join our existing [wiki:Mirrors list of mirrors]. Making MacPorts content available in more locations makes MacPorts faster.

If you find inaccuracies in these instructions or have other comments or questions, please let us know by contacting [mailto:portmgr@macports.org portmgr].

[[PageOutline(2-3,Table of Contents,inline)]]

== Contact us

First, please contact [https://lists.macports.org/mailman/listinfo/macports-mgr portmgr] to let us know you would like to create a mirror. Please tell us the following information:

* The name and web page URL of the organization providing the mirror, so that we can credit you properly.
* The city, state/province and [wikipedia:ISO_3166-1_alpha-2 ISO 3166-1 alpha-2 country code] where your mirror is located and the [wikipedia:International_Air_Transport_Association_airport_code IATA airport code] of the closest airport. The airport code and country code will become your MacPorts mirror name.
* GPS coordinates of your mirror's approximate location that we may display on a map. You may provide the coordinates of the above airport if you don't want to be more specific.
* Your mirror's existing hostname. We will create new hostnames abc.xy.distfiles.macports.org, abc.xy.packages.macports.org and abc.xy.rsync.macports.org (where "abc" is your airport code and "xy" is your country code) `CNAME`d to your existing hostname.
* Whether your server can be accessed over only http or only [#SSL https] or both.
* The email address of the server's administrator. We will list this on our mirrors page and MacPorts users might report problems to this address.
* Your mirror's upstream Internet connection speed and whether or not your mirror supports IPv6. We will list this information on our mirrors page.

We will review your request to become a mirror and will let you know if there are any problems. Meanwhile you can follow the rest of these instructions to get your mirror ready. We reserve the right to refuse a request to become a mirror for any reason, or to remove existing mirrors from use at any time and without notice.

If in the future you wish to temporarily or permanently discontinue providing your mirror, please let us know so that we can make the appropriate changes.

== Requirements

Running a MacPorts mirror will use a possibly significant amount of bandwidth. One of our servers in California, for example, served around 250GB of data via http and another 70GB of data via rsync per day. For http traffic, MacPorts chooses the closest mirror automatically, so how much traffic your mirror sees depends on the number of MacPorts users for whom your mirror is the closest. Ensure you have the permission of your network provider to run such a mirror, and that this amount of traffic will not be a problem for you.

You'll need a server with at least 2 TB of available disk space on a case-sensitive filesystem. You'll need a web server on port 80 such as Apache httpd or nginx on which you can configure new virtual hosts, and the rsync daemon running on port 873. The server should have a hostname and a static IPv4 address. IPv6 access is welcomed but is not required. The server must respond to pings. The server does not need to be running macOS and does not need to have MacPorts installed.

== Rsync modules

We recommend you copy all of our content by mirroring our ''macports'' rsync module, and we recommend you republish this on your server in a ''macports'' rsync module. If you prefer to copy only a subset of our content, you can mirror the packages, distfiles, release and/or trunk subdirectories of the ''macports'' module (into a ''macports'' rsync module on your server).

**Note:** We also offer those four directories as separate rsync modules, however they are deprecated. Please use only the ''macports'' module.

The four directories contain the following data:

* **packages** contains precompiled archives of each MacPorts port for several macOS versions. This is the largest directory, currently around 1.2 TB. Mirroring this data will provide the greatest benefit to MacPorts users near you since it lets them automatically receive these large files more quickly.
* **distfiles** contains the source code tarballs from which the precompiled archives were created. There are some ports that cannot legally be distributed as binaries, so those must build on the user's computer using these source files. This is the second-largest directory, currently around 700 GB. **Note:** this used to be called ''mpdistfiles''. If you are mirroring ''mpdistfiles'', please switch to mirroring ''distfiles''. If your local module is called ''mpdistfiles'', please rename it to ''distfiles'' and let us know so that we can update mirror_sites.tcl.
* **release** contains the latest released version of the MacPorts base source code which users update by running `sudo port selfupdate`, as well as the collection of Portfiles—the text files MacPorts reads that tell it how to install each port—which users update by running `sudo port sync`. Users near you can manually configure MacPorts to user your mirror for faster speed. This small directory has a size of less than 2 GB.
* **trunk** contains the latest development version of the MacPorts base source code. This small directory is around 120 MB.

== Get the files

You'll get the files from our server via [https://rsync.samba.org/ rsync]. You should use the latest version of rsync.

For your initial sync, you might speed up your transfer by using the nearest of the existing [wiki:Mirrors MacPorts mirror servers].
For subsequent scheduled syncing, use the master rsync server, rsync-origin.macports.org. 

**Note:** If you are syncing from a different hostname such as rsync.macports.org, or from a specific IP address, please switch to rsync-origin.macports.org.

There is a sample script below for mirroring these modules via rsync. You don't have to use that script if you already have other arrangements for syncing, but please use the following `rsync` flags, for the following reasons:

||=`rsync` flag =|| \
||=Purpose
|| [[span(`--compress`, style=white-space: pre)]], [[span(`-z`, style=white-space: pre)]] || \
|| File data is compressed during transfer. The server won't re-compress files that are already compressed.
|| [[span(`--delete-delay`, style=white-space: pre)]] || \
|| Files which were removed on the server are removed from the mirror after the transfer is finished. This option is similar to but more efficient than `--delete-after`.
|| [[span(`--hard-links`, style=white-space: pre)]], [[span(`-H`, style=white-space: pre)]] || \
|| Hard links are preserved.
|| [[span(`--links`, style=white-space: pre)]], [[span(`-l`, style=white-space: pre)]] || \
|| Symbolic links (symlinks) are preserved.
|| [[span(`--perms`, style=white-space: pre)]], [[span(`-p`, style=white-space: pre)]] || \
|| File and directory permissions are preserved.
|| [[span(`--recursive`, style=white-space: pre)]], [[span(`-r`, style=white-space: pre)]] || \
|| Directories are copied recursively.
|| [[span(`--timeout=600`, style=white-space: pre)]] || \
|| If no data is transferred for 10 minutes, the connection is dropped.
|| [[span(`--times`, style=white-space: pre)]], [[span(`-t`, style=white-space: pre)]] || \
|| Modification times are preserved.

== Stay up to date

After the initial sync is complete, keep your mirror up to date by periodically re-running the sync script. You can do this using cron, launchd, systemd, or another scheduler. A sample cron configuration is provided below.

The sample configuration syncs every hour. It is not necessary to sync that often, but MacPorts developers commit dozens of port updates each day, so the more frequently you sync, the more likely your mirror is to have the content users are requesting. Ensure your mirror syncs at least once daily.

== Rsync server

MacPorts retrieves files from ''release'' and ''trunk'' via rsync. MacPorts comes preconfigured to use the server rsync.macports.org but users can edit their configuration to use your mirror instead. In addition, other mirror administrators may wish to copy your ''packages'' and ''distfiles''. So you should run an rsync daemon to republish the files you synced.

The MacPorts rsync modules have generic names, so we suggest you put our files in subdirectories of a single "macports" module. We would prefer you not rename those subdirectories. If your site convention is to use the original server's hostname as the rsync module name, please use the name ''rsync.macports.org'' (even though you are actually syncing from rsync-origin.macports.org).

**Note:** If you are providing the modules under nonstandard names, please edit your rsyncd configuration to provide the files under a single module with subdirectories, but continue to serve the files from your old module names so that existing users can still find the files. If you are already providing a single module but with nonstandard subdirectory names, rename the subdirectories to their standard names but add symlinks at their previous names. Edit the [wiki:Mirrors Mirrors] page to reflect your changes, and if you changed your configuration for ''packages'' or ''distfiles'' let us know so that we can update mirror_sites.tcl and/or archive_sites.tcl.

There is a sample rsyncd configuration below.

== Web server

MacPorts retrieves files from ''packages'' and ''distfiles'' via http or https. MacPorts automatically downloads these files from whichever mirror is closest. So you should run a web server to make these modules available.

The contents of the ''packages'' module should be made available directly under !http(s)://abc.xy.packages.macports.org/ and the contents of the ''distfiles'' module should be made available directly under !http(s)://abc.xy.distfiles.macports.org/. It is not necessary to provide the contents of the ''release'' or ''trunk'' modules via http.

**Note:** If you are currently providing the contents of the ''packages'' or ''distfiles'' modules in a subdirectory of your MacPorts mirror hostname, you should reconfigure your web server to provide the contents directly under the MacPorts mirror hostname, and let us know so we can update archive_sites.tcl and/or mirror_sites.tcl. You should configure HTTP 301 redirects from the previously-used URLs to the new ones.

Directory listings should be turned on. URL spelling correction and multiviews must be turned off. There are sample web server configurations below.

== SSL

MacPorts verifies the integrity of downloaded files via checksums or signatures so your mirror is not required to provide access via https, but you can if you wish. [https://letsencrypt.org Let's Encrypt] offers free SSL/TLS certificates which can be used. Please add your `abc.xy.(distfiles|packages).macports.org` hostnames as Subject Alternative Names in your SSL/TLS certificate.

We suggest you continue to provide access via http, even if you also provide access via https. MacPorts still supports very old OS versions such as Mac OS X 10.4 Tiger, and the version of OpenSSL in old OS versions is not able to communicate with modern SSL web servers. (The cutoff version depends on which encryption algorithms you've configured your web server to allow.) You can disallow http access if you wish, but if you do so it will limit the OS versions that are able to connect to your server.

== Add mirror to MacPorts

Once your server is synced and ready to accept MacPorts traffic, let us know and we will edit archive_sites.tcl and mirror_sites.tcl to add the URLs of your MacPorts mirror hostnames. Also edit the [wiki:Mirrors Mirrors] page to add information about your server, or we can add it for you.

== Sample scripts and configuration files

=== Sample rsync update script

This sample script can be used to mirror all MacPorts content to your server using rsync.

For your initial sync, you can possibly make your transfer faster by uncommenting the `RSYNC_URL` line for a mirror server closer to you. For subsequent scheduled syncs, use the origin server to ensure you get the latest content.

{{{
#!/bin/bash

# This script mirrors all MacPorts content from RSYNC_URL to RSYNC_DIR.

set -euo pipefail

RSYNC_DIR="/var/www/html/macports"

RSYNC_URL="rsync://rsync-origin.macports.org/macports/"
#RSYNC_URL="rsync://jnb.za.rsync.macports.org/macports/"
#RSYNC_URL="rsync://jog.id.rsync.macports.org/macports/"
#RSYNC_URL="rsync://lil.fr.rsync.macports.org/macports/"
#RSYNC_URL="rsync://nue.de.rsync.macports.org/macports/"
#RSYNC_URL="rsync://osl.no.rsync.macports.org/macports/"

RSYNC="rsync"
RSYNC_ARGS="--compress --delete-delay --hard-links --links --no-motd --perms --recursive --stats --timeout=600 --times"
if [ -t 1 ]; then
    RSYNC_ARGS="$RSYNC_ARGS --info=progress2"
fi

echo "Syncing from $RSYNC_URL to $RSYNC_DIR"
$RSYNC $RSYNC_ARGS "$RSYNC_URL" "$RSYNC_DIR"
echo
}}}

If you wish to mirror only some of our content, you can instead use the following script and remove unwanted modules from the RSYNC_MODULES line.

{{{
#!/bin/bash

# This script mirrors the selected RSYNC_MODULES from RSYNC_URL to RSYNC_DIR.

set -euo pipefail

RSYNC_DIR="/var/www/html/macports"

RSYNC_MODULES="packages distfiles release trunk"

RSYNC_URL_FORMAT="rsync://rsync-origin.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://cjj.kr.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://fco.it.rsync.macports.org/macports-%s/"
#RSYNC_URL_FORMAT="rsync://jnb.za.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://jog.id.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://lil.fr.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://mse.uk.rsync.macports.org/%s.macports.org/"
#RSYNC_URL_FORMAT="rsync://nou.nc.rsync.macports.org/macports/%s.macports.org/"
#RSYNC_URL_FORMAT="rsync://nue.de.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://osl.no.rsync.macports.org/macports/%s/"
#RSYNC_URL_FORMAT="rsync://sea.us.rsync.macports.org/%s/"

RSYNC="rsync"
RSYNC_ARGS="--compress --delete-delay --hard-links --links --no-motd --perms --recursive --stats --timeout=600 --times"
if [ -t 1 ]; then
    RSYNC_ARGS="$RSYNC_ARGS --info=progress2"
fi

for RSYNC_MODULE in $RSYNC_MODULES; do
    RSYNC_URL="$(printf "$RSYNC_URL_FORMAT" "$RSYNC_MODULE")"
    echo "Syncing from $RSYNC_URL to $RSYNC_DIR/$RSYNC_MODULE"
    $RSYNC $RSYNC_ARGS "$RSYNC_URL" "$RSYNC_DIR/$RSYNC_MODULE"
    echo
done
}}}

=== Sample cron configuration

This sample cron configuration runs the above sync script every hour at 30 minutes past the hour:

{{{
30 * * * * /usr/local/bin/macports-rsync.sh 2>&1 >> /var/log/macports-rsync.log
}}}

So that not all mirrors are trying to connect to the origin server at the same time, please change "30" to a random number between 0 and 59.

=== Sample Apache httpd configuration

This sample configuration can be used to serve MacPorts files with Apache httpd.

{{{
<Directory /var/www/html/macports>
    Order allow,deny
    Allow from all
    AllowOverride None
    Options +Indexes -FollowSymLinks -MultiViews
    IndexOptions Charset=UTF-8 FancyIndexing IconsAreLinks HTMLTable NameWidth=* SuppressDescription
    AddType application/mac-binhex40 .hqx
    AddType application/octet-stream .bin .exe .rmd160
    AddType application/vnd.apple.installer+xml .dist .distz .mpkg .pkg
    AddType application/x-7z-compressed .7z
    AddType application/x-apple-diskimage .dmg .image .img .smi
    AddType application/x-bzip2 .bz2 .tbz .tbz2
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType application/x-iso9660-image .iso
    AddType application/x-lzip .lz
    AddType application/x-lzma .lzma .tlz
    AddType application/x-rar-compressed .rar
    AddType application/x-stuffit .sit
    AddType application/x-stuffitx .sitx
    AddType application/x-tar .tar
    AddType application/x-tcl .tcl
    AddType application/x-xz .txz .xz
    AddType application/zip .zip
    AddType image/icns .icns
</Directory>
<VirtualHost *:80>
    ServerName abc.xy.distfiles.macports.org
    ServerAlias distfiles.macports.org
    DocumentRoot /var/www/html/macports/distfiles
    UseCanonicalName On
    <IfModule mod_speling.c>
        CheckSpelling Off
    </IfModule>

    Header set "Cache-Control" "max-age=2592000, public"
    <LocationMatch "/$">
        Header set "Cache-Control" "max-age=300, public"
    </LocationMatch>
    <LocationMatch "^/ports\.tar\.gz$">
        Header set "Cache-Control" "max-age=1800, public"
    </LocationMatch>
</VirtualHost>
<VirtualHost *:80>
    ServerName abc.xy.packages.macports.org
    ServerAlias packages.macports.org
    DocumentRoot /var/www/html/macports/packages
    UseCanonicalName On
    <IfModule mod_speling.c>
        CheckSpelling Off
    </IfModule>

    Header set "Cache-Control" "max-age=2592000, public"
    <LocationMatch "/$">
        Header set "Cache-Control" "max-age=300, public"
    </LocationMatch>
</VirtualHost>
}}}

If you are serving MacPorts files from subdirectories of your MacPorts mirror hostnames, use `Redirect` directives to redirect those requests to their new locations. For example, these directives show how to redirect requests for !http://hostname/pub/macports/packages/... to !http://abc.xy.packages.macports.org/... and requests for !http://hostname/pub/macports/distfiles/... to !http://abc.xy.distfiles.macports.org/...:

{{{
    Redirect 301 /pub/macports/packages http://abc.xy.packages.macports.org
    Redirect 301 /pub/macports/distfiles http://abc.xy.distfiles.macports.org
}}}

Place these directives inside the `VirtualHost` directives for your MacPorts mirror hostnames.

=== Sample nginx configuration

This sample configuration can be used to serve MacPorts files with nginx.

{{{
types {
    application/mac-binhex40 hqx;
    application/octet-stream bin exe rmd160;
    application/vnd.apple.installer+xml dist distz mpkg pkg;
    application/x-7z-compressed 7z;
    application/x-apple-diskimage dmg image img smi;
    application/x-bzip2 bz2 tbz tbz2;
    application/x-compress Z;
    application/x-gzip gz tgz;
    application/x-iso9660-image iso;
    application/x-lzip lz;
    application/x-lzma lzma tlz;
    application/x-rar-compressed rar;
    application/x-stuffit sit;
    application/x-stuffitx sitx;
    application/x-tar tar;
    application/x-tcl tcl;
    application/x-xz txz xz;
    application/zip zip;
    image/icns icns;
}
server {
    listen 80;
    listen [::]:80;
    server_name abc.xy.distfiles.macports.org distfiles.macports.org;
    root /var/www/html/macports/distfiles;
    server_name_in_redirect on;
    autoindex on;

    add_header Cache-Control "max-age=2592000, public";
    location ~ /$ {
        add_header Cache-Control "max-age=300, public";
    }
    location = /ports.tar.gz {
        add_header Cache-Control "max-age=1800, public";
    }
}
server {
    listen 80;
    listen [::]:80;
    server_name abc.xy.packages.macports.org packages.macports.org;
    root /var/www/html/macports/packages;
    server_name_in_redirect on;
    autoindex on;

    add_header Cache-Control "max-age=2592000, public";
    location ~ /$ {
        add_header Cache-Control "max-age=300, public";
    }
}
}}}

If you are serving MacPorts files from subdirectories of your MacPorts mirror hostnames, use `return` directives inside a `location` directive to redirect those requests to their new locations. For example, these directives show how to redirect requests for !http://hostname/pub/macports/packages/... to !http://abc.xy.packages.macports.org/... and requests for !http://hostname/pub/macports/distfiles/... to !http://abc.xy.distfiles.macports.org/...:

{{{
    location ^~ /pub/macports/packages(/.*)?$ {
        return 301 http://abc.xy.packages.macports.org$1;
    }
    location ^~ /pub/macports/distfiles(/.*)?$ {
        return 301 http://abc.xy.distfiles.macports.org$1;
    }
}}}

Place these directives inside the `server` directives for your MacPorts mirror hostnames.

=== Sample rsyncd configuration

This sample rsyncd.conf file can be used to serve MacPorts files with the rsync daemon.

{{{
dont compress = 7z ace avi bin bz2 deb distz dmg gem gif gpg gz \
    icns image img iso jar jpeg jpg lz lzma lzo mov mp3 mp4 mpkg msi \
    ogg pdf pkg png rar rmd160 rpm rzip sig sit sitx smi tbz tbz2 \
    tgz tlz ttf txz xz Z zip
forward lookup = false
gid = nobody
list = true
log file = /var/log/rsyncd/rsyncd.log
max connections = 0
pid file = /var/run/rsyncd.pid
read only = true
reverse lookup = false
timeout = 600
transfer logging = true
uid = nobody
use chroot = no

[macports]
    comment = MacPorts release, trunk, distfiles and packages
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports
}}}

Start rsync with:

{{{
rsync --daemon --config=/etc/rsyncd.conf
}}}

If you are serving individual rsync modules, move those directories into a single new ''macports'' module but continue to serve the old modules under your existing names so that users using the old names can still get the files. For example:

{{{
cd /var/www/html
mkdir macports
mv packages distfiles release trunk macports/
}}}

Adjust your rsyncd.conf to the new directory locations:

{{{
[release]
    comment = MacPorts release (deprecated; use "macports/release" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/release

[trunk]
    comment = MacPorts trunk (deprecated; use "macports/trunk" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/trunk

[distfiles]
    comment = MacPorts distfiles (deprecated; use "macports/distfiles" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/distfiles

[mpdistfiles]
    comment = MacPorts distfiles (deprecated; use "macports/distfiles" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/distfiles

[packages]
    comment = MacPorts packages (deprecated; use "macports/packages" instead)
    log file = /var/log/rsyncd/macports.log
    path = /var/www/html/macports/packages
}}}

If you are already serving a single module but with subdirectories with nonstandard names, move the subdirectories to their standard names and add a symlink so that users using the old nonstandard names can still get the files. For example, if you are currently serving ''mpdistfiles'', rename it to ''distfiles'' and add an ''mpdistfiles'' symlink:

{{{
cd /var/www/html/macports
mv mpdistfiles distfiles
ln -s distfiles mpdistfiles
}}}