Sharing Archives under MacPorts 2
- Audience: Users who want to share binary archives between machines
- Requires: MacPorts >= 2.0.0
Introduction
Users with multiple machines can save significant compilation effort by installing pre-built ports from binary archives that have been assembled on another system. Under previous MacPorts releases this involved enabling archive mode on all systems, manually transferring archives, and sometimes manually unpacking the archive before proceeding with the installation. MacPorts 2.0.0 introduces features which permanently enable archive mode and includes support for fetching archives from local or remote sources.
Installation
Note, this HowTo assumes a MacPorts installation prefix of /opt/local. If you have installed to an alternate location, please update the paths accordingly.
Also, the installation steps will refer to build and installation systems. The build system compiles, signs, and shares ports and archives. Each installation system retrieves the archives and installs the port, without having to compile anything.
Configuration Notes
MacPorts 2.10 introduces the portimage_mode setting in macports.conf. If you are going to share archives as described in this guide, the value must be set to archive or directory_and_archive. The default on filesystems that support cloning, such as APFS, is directory and the install phase will not create shareable archives.
Step 1: Generate signing keys
Before MacPorts will install software from a binary archive, it will test that the package has a valid signature that is locally trusted. For now, we just need to generate the keys and sign the packages.
The public and private keys are generated as described below.
openssl genrsa -des3 -out local-privkey.pem 2048 openssl rsa -in local-privkey.pem -pubout -out local-pubkey.pem
You will be prompted to enter a passphrase to protect the private key. It is up to you to decide how complex this protection should be. Note that it is possible to store this key and passphrase in your Keychain.app (using /usr/bin/ssh-add). This has the advantage of passing the key security on to the OS, though it may not be an acceptable option for automating package signing.
It is also possible to create a key without a passphrase. If you are not distributing packages outside of a home network, this is likely not a problem. An unprotected private key is generated as below; the command to generate the public key is unchanged.
openssl genrsa -out local-privkey.pem 2048
The passphrase can also be stripped from an existing private key using the following:
openssl rsa -in local-privkey.pem -out newlocal-privkey.pem
The public and private keys can be stored anywhere as long as they are accessible to the following steps. This tutorial has them placed on the build system at /opt/local/share/macports/.
Note: While the build system need only have access to the private key, it's a good idea to store both the public and private keys in the same location on the build system for safe keeping. The installation system should only have access to the public key.
Step 2: Let's sign some packages
Now that the keys are generated, we can test signing a package. Any file will do; the following example uses archive.tbz. Your example should specify the full path to the private key, the archive, and the archive signature that should be named identically to the input archive and suffixed with .rmd160.
openssl dgst -ripemd160 -sign local-privkey.pem -out archive.tbz2.rmd160 archive.tbz2
You can verify the signature as well:
openssl dgst -ripemd160 -verify local-pubkey.pem -signature archive.tbz2.rmd160 archive.tbz2
Signing an archive for the lighttpd port might look something like this:
openssl dgst -ripemd160 \
-sign /opt/local/share/macports/local-privkey.pem \
-out /opt/local/var/macports/software/lighttpd/lighttpd-1.4.28_0+ssl+universal.darwin_10.i386-x86_64.tbz2.rmd160 \
/opt/local/var/macports/software/lighttpd/lighttpd-1.4.28_0+ssl+universal.darwin_10.i386-x86_64.tbz2
Now, signing every individual archive like this would be quite time consuming. It'd be better to run a script which finds and signs every archive. Such a script is displayed below:
#!/bin/sh
PRIVKEY="/opt/local/share/macports/local-privkey.pem"
PUBKEY="/opt/local/share/macports/local-pubkey.pem"
SOFTWARE="/opt/local/var/macports/software"
# First, clear out any outdated signatures
for SIGNATURE in "$SOFTWARE"/*/*.rmd160
do
ARCHIVE_DIR="$(dirname "$SIGNATURE")"
ARCHIVE="$(basename "$SIGNATURE" .rmd160)"
if [ "$SIGNATURE" -ot "$ARCHIVE_DIR"/"$ARCHIVE" -o ! -f "$ARCHIVE_DIR"/"$ARCHIVE" ]
then
/bin/echo removing outdated signature: "$SIGNATURE"
/bin/rm -f "$SIGNATURE"
fi
done
# Now, find every archive that doesn't have a signature
for ARCHIVE in "$SOFTWARE"/*/*.{tbz2,tgz,tar,tbz,tlz,txz,xar,zip,cpgz,cpio}
do
PORTNAME="$(basename "$(dirname "$ARCHIVE")")"
ANAME="$(basename "$ARCHIVE")"
if [ "$ARCHIVE" -nt "$ARCHIVE".rmd160 ]
then
/bin/echo -n signing archive: "$ANAME "
/usr/bin/openssl dgst -ripemd160 -sign "$PRIVKEY" -out "$ARCHIVE".rmd160 "$ARCHIVE"
/usr/bin/openssl dgst -ripemd160 -verify "$PUBKEY" -signature "$ARCHIVE".rmd160 "$ARCHIVE"
fi
done
The script can be run after upgrading, installing, or uninstalling ports to remove outdated signatures and generate updated signatures for all available archives.
Step 3: Share archives
In order to retrieve archives from the build system, they must first be shared. MacPorts expects this to be a basic web listing of the software directory (/opt/local/var/macports/software/). While Mac OS X ships with an installation of Apache, that's a bit heavy for what we need. Instead, the light weight web server lighttpd can be used with minimal configuration.
Conveniently, it's available through MacPorts, so go ahead and install that.
port install lighttpd
Create the configuration file and save it next to the public and private keys. (Like the keys, this file can be stored anywhere that is accessible.)
server.document-root = "/opt/local/var/macports/software/"
server.username = "_www"
server.groupname = "_www"
server.port = 6227
dir-listing.activate = "enable"
mimetype.assign = (
".tbz2" => "application/x-bzip-compressed-tar",
".rmd160" => "text/binary",
# make the default mime type application/octet-stream.
"" => "application/octet-stream",
)
Note: The server.port configuration is somewhat arbitrary as long as something non-standard is used so as not to collide with an existing service. The standard ports 80 or 8080 are options if you are not running another web server.
You should now be able to start the service...
/opt/local/sbin/lighttpd -D -f /opt/local/share/macports/macports-archives-lighttpd.conf
and view your software archives in a browser: http://localhost:6227
There is also a simple tutorial for enabling SSL support for additional security if it is desired. Be sure to install lighttpd with the +ssl variant.
You'll also want the lighttpd service to run at all times so your client machines are not deprived of access to the port archives. A simple LaunchD task will handle this:
/Library/LaunchDaemons/org.macports.lighttpd.share-archives.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>org.macports.lighttpd.share-archives</string>
<key>LowPriorityIO</key>
<true/>
<key>Nice</key>
<integer>1</integer>
<key>ProgramArguments</key>
<array>
<string>/opt/local/sbin/lighttpd</string>
<string>-D</string>
<string>-f</string>
<string>/opt/local/share/macports/macports-archives-lighttpd.conf</string>
</array>
<key>StandardErrorPath</key>
<string>/var/log/macports-archives-lighttpd.out</string>
<key>StandardOutPath</key>
<string>/var/log/macports-archives-lighttpd.out</string>
</dict>
</plist>
Make sure your test server has been killed and then load the task with:
launchctl load /Library/LaunchDaemons/org.macports.lighttpd.share-archives.plist
Ensure that the server is running again by browsing the page as before.
Step 4: Fetch archives
Finally we're ready to actually fetch some archives.
First, copy the public key to each client system that will be installing from archives. As before, /opt/local/share/macports/ is a decent enough location.
Now, MacPorts needs to be configured to fetch archives from the build system. The archive_site_local setting can be set to an IP, but it will likely be more convenient to use the Bonjour name of the local system. This name can be found in the Sharing preference pane.
Add the following to /opt/local/etc/macports/archive_sites.conf
name bonjour urls http://bonjour.local:6227/
A line indicating the location of the public key must also be added to /opt/local/etc/macports/pubkeys.conf. Something like:
/opt/local/share/macports/local-pubkey.pem
And you're done! You should now be able to compile ports once on your build system and install on as many client systems as you like (as long as the requested variants and architectures are the same).