Continuous Integration of KDE software on MacPorts

Currently there is an ongoing discussion on KDE-DEVEL about this topic. MacPort's key developers' expertise regarding the setup of their buildbots could perhaps be helpful for the KDE folks. On the other hand one might also run Jenkins on OSX and adapt its configuration to the KDE software's requirements.

Requirements for the CI system

This is an excerpt from ​Ben Cooksley's post on KDE-DEVEL on thread "Running KDE apps on Apple OS X":

---

The ​infrastructure we have for our Linux builds should be nearly completely portable to OSX without too much trouble (in theory at least - i've never done any compilation on OSX).

The actual requirements aren't written anywhere, but you might find the ​documentation I've written for the CI system to be of some use.

A basic run down of what the CI system needs however:

1) Python 2.x, with json and lxml support (for the scripts which conduct builds)

2) Java (for the Jenkins node agent itself)

3) RSync and SSH (for the transferral of completed builds between nodes)

4) Git, Mercurial, Subversion, Bazaar, wget and GNU Tar (to access source code)

5) GNU Patch (for applying custom patches, used in certain builds)

6) A compiler, usable by CMake, QMake and autotools based build systems

7) GNU Make, Automake, Autoconf (for carrying out the build, and configuring it in certain rare cases)

We'll need to make adjustments to ensure the system doesn't attempt to launch Xvfb or a X11 Window Manager, which it currently will do when executing tests. These should be fairly easy to do however.

To build certain projects, the compiler will need to support C# and a certain level of C++11 as well. The Mono bindings will not be buildable if C# support is unavailable.

Under no circumstance should a CI node have Qt installed at the system level in any form, as this is provided directly by the CI system itself.

---

Checking on the above requirements in the current MacPorts tree shows that all necessary tools are available.
Which versions of these programs can be used with KDE's current CI system has to be checked step by step.

The only problem seems to be C# support. But those projects will probably only be running on Windows anyways.

Setting up the CI system (still hanging on OSX up to now)

This is what has to be done to get a KDE/CI system up and running on OSX (partially by also using MacPorts):

A) Start with fresh Mavericks install and make sure your disk has at least 20G free space left

B) Install MacPorts 2.2.1 using the official PKG

C) Install ports for bash use:

$ sudo port install bash bash-completion

D) Set up ​bash environment for MacPorts use regarding search paths (PATH, MANPATH)

E) Install Xcode from AppStore (and agree actively to the developer license agreement)

F) Install developer command line tools and install some of MacPorts' ports needed for later

$ xcode-select --install
$ sudo port install py27-lxml bazaar mercurial wget automake autoconf

Installing bazaar brings the also needed port gnutar into the system!

G) Since some tools are already installed on OSX due to Xcode (like git, svn, ssh, rsync, make, clang etc.) they don't need to be installed via MacPorts for now

H) Create some folders, clone and update KDE's CI system (assuming a user "kdeci" with its home directory "/Users/kdeci"):

$ cd
$ mkdir -p WC/KDECI-build;
$ cd WC
$ git clone git://anongit.kde.org/websites/build-kde-org
$ cd build-kde-org
$ git checkout production
$ ./update-setup.sh                    # Ignore 404 occurring due to error cloning ECMA262 by hg

The update shell script created a folder ~/scripts in which subsequent commands will have to be executed.

I) Prepare environment for Qt5 build (set correct path to kde install directory in config/build/darwin-mavericks.cfg):

$ sudo port install dbus libmng libpng mysql55 pkgconfig sqlite2 tiff
$ sudo port uninstall cmake            # Remove MacPorts' cmake (which came into the system as build-dependency for mysql55)
$ cd ~/scripts
$ cat config/build/darwin-mavericks.cfg
[DEFAULT]
scriptsLocation=/Users/kdeci/scripts
opSys=darwin
architecture=mavericks
compiler=clang

[General]
installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/
$ python2.7 tools/prepare-environment.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/qt5
...
$

J) Set up build environment:

$ cd ~/scripts; cat config/build/qt5/darwin-mavericks.cfg
[Build]
configureCommand=%(configureExecutable)s -release -system-zlib -system-libpng -system-libjpeg -system-sqlite -dbus -plugin-sql-mysql -nomake examples -confirm-license -opensource -prefix {instPrefix} -no-framework -v -I/opt/local/include -L/opt/local/lib

Consider here the use of options "-debug -separate-debug-info" instead of "-release".

K) Include MySQL's binary path into PATH (assuming /opt/local as MacPort's prefix):

export PATH=/opt/local/lib/mysql55/bin:$PATH

L) Set up SSH for rsync access to build host, then start building Qt5:

$ cat ~/.ssh/config 
Host build.kde.org
   Port 2022
$ ssh-keygen -C YOUR@EMAIL.ADDRESS       # <-- send your public SSH key to Ben Cooksley in order to get access to the build server
$ python2.7 tools/perform-build.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/qt5
...

M) Proceed with installation of cmake (needs manual checkout of sources):

$ python2.7 tools/prepare-environment.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/cmake
...
$ git clone git://cmake.org/cmake.git ~/WC/KDECI-build/cmake
$ python2.7 tools/perform-build.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/cmake
...

N) Installation of KDE frameworks, starting with kauth:

N.1) Framework kauth:

$ python2.7 tools/prepare-environment.py --project kauth --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/kauth
...
$ python2.7 tools/perform-build.py --project kauth --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/kauth
...

N.2) Framework polkit-qt-1 (still fails due to inability to find package Polkit):

$ python2.7 tools/prepare-environment.py --project polkit-qt-1 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/polkit-qt-1
...
$ python2.7 tools/perform-build.py --project polkit-qt-1 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/polkit-qt-1
...

OSX virtualisation

When setting up build-slaves it might be a good idea to not only have several Apple machines in real hardware, but perhaps also make use of some virtual OSX machines. This might speed up especially the test of various environments or give opportunity for an easy backup of the complete CI system.

See ​SOFTWARE LICENSE AGREEMENT FOR OS X MAVERICKS:

---

If OS X Mavericks is pre-installed “you are granted a limited, non-exclusive license to install, use and run one (1) copy of the Apple Software on a single Apple-branded computer at any one time”.

If OS X Mavericks is downloaded from the Mac App Store, “you are granted a limited, non-transferable, non-exclusive license (i) to download, install, use and run for personal, non-commercial use, one (1) copy of the Apple Software directly on each Apple-branded computer running OS X Mountain Lion, OS X Lion or OS X Snow Leopard (“Mac Computer”) that you own or control; … (iii) to install, use and run up to two (2) additional copies or instances of the Apple Software within virtual operating system environments on each Mac Computer you own or control that is already running the Apple Software, for purposes of: (a) software development; (b) testing during software development; (c) using OS X Server; or (d) personal, non-commercial use.”

---