Latest news

The new multi-platform KDE/CI system designed and implemented by Ben and Scarlett in her ​SOK project has finally gone live as a ​drop-in-replacing KDE's old Jenkins CI.

Current status of setting up a OSX/CI system for Qt5+KF5

This page shall describe what has to be done to get a CI system for Qt5+KF5 up and running on an OSX host. Partially this is achieved by using quite a few ports from MacPorts.

The same could and should be done for Qt4+KDE4, as KDE4 software is going to coexist with KF5 for a longer time. The only hurdle is, that one needs to try and patch Qt4 for the CI system in such a way that it installs on OSX fine (e.g. by integrating all patches applied on MacPorts' qt4-mac port.)

Preparing the OSX host system

Apple developer tools

Apple's developer tools are a necessary precondition for MacPorts as well as for the CI system. Basically you've got to follow the steps which are a must before you can install MacPorts itself by installing

• Xcode from the AppStore
• developer command line tools from the command line

  $ xcode-select --install
  

and agree actively to the developer license agreement

$ xcodebuild -license

and you're ready to go ahead.

Start with a fresh Mavericks/MacPorts install

The setup of an OSX/CI system has only been tested with OSX 10.9 (Mavericks). It is not known, whether this also works for older OSX versions.

Using the current official MacPorts version is recommended. To be on the safe side one needs to make sure that the disk has at least about 120G space in total, if it is planned to build both KF5 branch-groups 'kf5-qt5' and 'stable-kf5-qt5' with Qt 5.3.2 and 5.4.1, respectively. However, total disk consumption depends on which projects will have to be built by the OSX/CI system.

Getting a MacPorts-compatible bash environment

Install ports for MacPorts' bash

$ sudo port install bash bash-completion

and set up the ​bash environment for MacPorts use regarding search paths (PATH, MANPATH)

Tools already present on OSX

Some tools are already installed on OSX due to Xcode (like git, svn, ssh, rsync, make, clang etc.) they don't need to be installed via MacPorts for now. Should a more up-to-date version of any one of these be necessary for the CI system it could be added later.

Some more software supplied by MacPorts

Install ports needed for the CI scripts as well as KF5, e.g.:

$ # Ports for CI:
$ sudo port install gnutar py27-lxml bzr mercurial wget automake autoconf cppcheck lcov
$
$ # Ports for KF5:
$ sudo port install shared-mime-info docbook-xml docbook-xsl giflib boost

The full list is being kept in ​~/scripts/ports-list.txt, which can be installed in one go by

$ sudo port install `cat ports-list.txt | egrep -v "^(#.*|\w*)$" | xargs`

Setting up the CI system itself

Basic setup of CI system

In order to start building up a build slave for KDE's Jenkins-based CI system it is in principle needed to clone ​websites/build-kde-org, BUT further down it will become clear, that it is a good idea to track all changes to be made for websites/build-kde-org in a dedicated branch. That is why the ​dedicated repository clone "mp-osx-ci" has been introduced which has the branch 'mp-osx-ci' containing all changes needed for a CI system on OSX.

Create some folders, clone and update CI system (assuming a user "kdeci" with its home directory "/Users/kdeci"):

$ cd
$ git clone git://anongit.kde.org/clones/websites/build-kde-org/kaning/mp-osx-ci.git scripts
$ cd scripts
$ git checkout mp-osx-ci
$ ./update-setup.sh                    # Ignore 404 occurring due to error cloning ECMA262 by hg

The update shell script created a folder ~/scripts in which all subsequent commands will have to be executed.

(BTW, if you were checking out the original repository from ​git://anongit.kde.org/websites/build-kde-org you'd have to check out the "production" branch.)

Note: All configuration settings listed below should already be present in the configuration set up in branch 'mp-osx-ci'.

Automatic installation of a CI system

There is ongoing effort to create a script which allows an easy setup of an OSX/CI system from scratch. The script ​ci-install.sh is not yet complete, but can serve as a starting point for further work in this directions.

SSH setup

Set up SSH for rsync access to build host:

$ cat ~/.ssh/config 
Host build.kde.org
   Port 2022
$ ssh-keygen -C YOUR@EMAIL.ADDRESS       # <-- send your public SSH key to Ben Cooksley in order to get access to the build server

Helper bash scripts

KDE's Jenkins-based CI system mainly ​makes use of these two python scripts:

tools/prepare-environment.py
tools/perform-build.py

For qt5 they've further down been used in order to show the workflow. But in daily use it is more practical to consider some helper bash scripts which ease all actions needed to run project builds on an OSX/CI system for which there's not yet the Jenkins integration available.

The various projects on the CI system have to be handled for the branch-group 'kf5-qt5' by replacing "PROJECT" with the project name of the corresponding framework or application:

$ python2.7 tools/prepare-environment.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/PROJECT
...
$ (cd ~/WC/KDECI-builds/kf5-qt5/PROJECT; git checkout jenkins)
...
$ python2.7 tools/perform-build.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/PROJECT
...

or alternatively by using ​helper bash scripts in ~/scripts/ present in the mp-osx-ci branch of the mp-osx-ci repository

$ ./prepare.sh PROJECT
...
$ ./build.sh PROJECT
...

where prepare.sh takes care of creating the build directory if it is not yet existing as well as checks out the branch 'jenkins'. Alternatively one can combine these steps using:

$ ./install.sh LIST_OF_PROJECTS [rebuild]

The 'rebuild' option can be used to trigger calling build.sh even if prepare.sh didn't signal any changes in the remote project repository.

Please note that also here 'kf5-qt5' is the default branch-group! If one want to build for branch-group 'stable-kf5-qt5' one needs to create the file ~/scripts/branch-group.sh in order to define the variable 'BRANCH_GROUP' which will then be used by the bash helper scripts later on:

$ cat ~/scripts/branch-group.sh 
BRANCH_GROUP=stable-kf5-qt5

If you want to revert back to 'kf5-qt5' simply comment out the above definition or delete the file.

1st step: Qt5 build

Prepare environment for Qt5 build

We still need to deactivate two ports and set the correct path to KDE-install directory in config/build/darwin-mavericks.cfg:

$ # Further up ports for Qt should have been installed as listed in ports-list.txt:
$ #   sudo port install zlib openssl dbus jpeg tiff libmng libpng mysql55 pkgconfig sqlite3 icu pcre libiconv
$
$ sudo port deactivate cmake            # Remove MacPorts' cmake (which came into the system as build-dependency for mysql55) just to be on the safe side
$ sudo port deactivate libarchive       # Avoid https://bugreports.qt-project.org/browse/QTBUG-42605 -> https://bugreports.qt-project.org/browse/QTBUG-42870
$ cd ~/scripts
$ cat config/build/darwin-mavericks.cfg
[DEFAULT]
scriptsLocation=/Users/kdeci/scripts
opSys=darwin
architecture=mavericks
compiler=clang
configurePlatformArgs=-DCMAKE_INSTALL_BUNDLEDIR="Applications/KF5" -DCMAKE_DISABLE_FIND_PACKAGE_X11=ON

[General]
installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/

[Test]
testsEnabled=False

$ python2.7 tools/prepare-environment.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/qt5
...
$

BTW, it could be important to also install freetype for Qt5, but it hasn't been clarified yet whether it should be there or not.

Standard paths for Qt5

A very important issue for Qt5 and KF5 is that QStandardPaths gets set appropriately for OSX in general and especially for the OSX/CI system, as this installs all frameworks and apps into separate install roots. To enable Qt5/KF5 apps to find the correct locations for files (for configuration, read-only data, writeable data, etc.) it is necessary to patch QStandardPaths. This is done as a ​temporary hack in

~/scripts/patches/qt5/kf5-qt5/patch-qstandardpaths_mac.cpp.diff

and still requires more work. The patch enables kdoctools and kdelibs4support to locate their files successfully (which had up to now to be handled using an even nastier workaround), but apart from that there are still problems with config files. Thanks to Jeremy Whiting there is an ​ongoing effort to get a QSP patch into the Qt sources. In short: this is still work in progress...

More information about standard paths is given for ​Qt5 and ​KF5 as well as in ​Ian's post on KDE-MAC.

Note: Due to Qt's bug ​41136 there is ​another patch needed for Qt < 5.4.

Set up build environment for Qt5

It is required to build the debug version. Also make sure that all libs can be found by adding search paths, use "-dbus-linked" to make the ​session bus work and avoid inclusion of MacPorts' glib2:

$ cd ~/scripts; cat config/build/qt5/darwin-mavericks.cfg
[Build]
configureCommand=%(configureExecutable)s -debug -separate-debug-info -system-zlib -system-libpng -system-libjpeg -system-sqlite -dbus-linked -plugin-sql-mysql -nomake examples -confirm-license -opensource -prefix {instPrefix} -no-framework -v -I/opt/local/include -L/opt/local/lib -no-glib

Include MySQL's binary path into PATH e.g. in .macports/profile (assuming /opt/local as MacPort's prefix):

export PATH=/opt/local/lib/mysql55/bin:$PATH

Start the qt5-build

$ python2.7 tools/perform-build.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/qt5
...

Installation of cmake

CMake needs a manual checkout of its sources (which is demonstrated here the explicit way again):

$ python2.7 tools/prepare-environment.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/cmake
...
$ git clone git://cmake.org/cmake.git ~/WC/KDECI-builds/kf5-qt5/cmake
$ python2.7 tools/perform-build.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/cmake
...
$ 
$ # Alternatively you can simply use this script (which does all of the above for you):
$ ./install.sh cmake

Important note:

• For a while it seemed to be necessary to NOT to install the latest cmake version on OSX, as this causes ​build problems for framework kcoreaddons later on. However, the commit 3c0d87645f5861e9fd6fd78855697f77ef77144e was recommended by Cristian Onet which worked fine.

• In October 2014 the same regression reappeared, but was fixed in version 3.1.0-rc1

• The latest cmake version (being thoroughly tested against KF5) is secretly being set in prepare.sh.

Setting up the build environment

The branch 'mp-osx-ci' contains all required configuration settings for OSX. The most important shall be pointed out here:

• For KF5 on OSX one has to modify the configuration settings by commenting out the kauth dependency on polkit-qt-1, since that dependency is only needed on Linux.

• Problems appeared for some frameworks since kconfig_compiler_kf5.app doesn't yet get installed on OSX (the build scripts error out with "/Applications/KDE/kconfig_compiler_kf5.app/Contents/MacOS/kconfig_compiler_kf5 as location is wrong"). This can be fixed for now introducing a temporary configuration file for kconfig which sets CMAKE_INSTALL_BUNDLEDIR to make sure that the application bundle gets installed below PREFIX/lib/libexec/.

• CMAKE_INSTALL_BUNDLEDIR and DATA_INSTALL_DIR need to be set for OSX-like directories. (NOT USED AT THE MOMENT, since all files currently get installed just like on Linux, as long as the ​QSP patch is still not finalized.)

• Disable searching for X11 by setting CMAKE_DISABLE_FIND_PACKAGE_X11=ON.

• Disable running tests for now, in order to save time during rebuilding.

• Coverage doesn't work for kdevplatform and plasma-framework

The file kde_projects.xml is necessary in order to run the CI scripts. Usually it should be kept up to date in ~/scripts, but in some cases it might be corrupted or not existing at all. Then one should download it from ​KDE-projects.

Keeping the OSX host system up-to-date

Updating the OSX operating system as well as the MacPorts installation should not be a problem for the host system as qt5 and all KF5 projects should be binary-compatible. In doubt, or if a project fails because of - indeed possible - library version changes (which is e.g. explicitly checked on MacPorts after installing new ports) simply rebuild all projects.

Jenkins integration

Using all-tiers-install.sh or the other bash scripts allows running the OSX/CI system manually.

However, in order to seamlessly integrate this into KDE's automatically functioning CI infrastructure it is necessary to create a ​Jenkins builder yaml configuration which will replace the currently used configuration on ​http://build.kde.org allowing to configure build jobs for other platforms like Windows and OSX.

This enormous project has been tackled in ​Scarlett G. Clark's SOK project and has eventually gone live by [drop-in-replacing KDE's old Jenkins CI ​https://build.kde.org].

Some experiments have been made to get a Jenkins slave for OSX up and running, which is still using the classic configuration approach. The changes needed for the two builder scripts are kept in ​branch 'mp-osx-ci-jenkins' in mp-osx-ci. At the moment only a few projects are configured, which can be built successfully, even the ​multi-variation project Trojita.

Installation of KF5 frameworks and other projects

Specific projects can be installed using install.sh, but one can also build all projects of a specific tier as a whole using

$ # This builds all frameworks of tier 1
$ ./tier-install.sh 1
...
$ # This builds all frameworks of tier 2, but forces rebuilding every project regardless of
$ # whether git pulled in changes or not (useful when e.g. qt5 or cmake was updated)
$ ./tier-install.sh 2 rebuild
...
$ # This rebuilds all frameworks of tier 3 starting at project kio (i.e. skipping all predecessor frameworks)
$ ./tier-install.sh 3 kio rebuild
...

All projects of all tiers can be build easily using

$ ./all-tiers-install.sh [ START_TIER [START_PROJECT] ] [rebuild]

(optionally) beginning at START_TIER building up to tier 5. Tiers 0 and 5 are just "virtual" tiers - which were introduced to more easily build some projects.

As of July 5th 2014 all KF5 frameworks build successfully.

Troubleshooting for projects' dependencies

​Dependency data for the CI system is being downloaded by ~/scripts/update-setup.sh to ~/scripts/dependencies/. This folder contains files defining the dependencies (see subsections below) as well as some tools.

Sometimes it might be needed to add a new entry to the KF5-specific (logical) dependency data in case a project cannot find a required framework, although it is already available on the CI system! Details concerning this can be found on a dedicated page.

Notes concerning some files installed by the frameworks

All notes regarding the placement of certain files (apps, config, data, etc.) have been moved aside to a dedicated page as the patching of Qt5's QStandardPaths is still work in progress.

KF5 software beyond the basic frameworks

Since the basic set of frameworks is now ready for use on OSX one can go ahead and start building other software depending on KF5. A first step is building kdesupport-svn which is a prerequisite for further projects.

Important projects are e.g. KDevelop, KMyMoney, but also ​KDE-EDU applications on their ​route to KF5 as well as ​KDE games can serve as good test cases. This is referred to as "Tier 5" and details can be found here.

Regularly updated information can be found on Christoph Feck's ​porting status page.

Some stats about the OSX/CI system

Altogether there are more than two hundred KF5 frameworks and applications being regularly built for OSX (as of March 14th 2015):

$ cat tier*.fw | egrep -v "^#" | wc | awk '{print $1}'
231

of which currently only 13 fail.

The roundtrip time for a full check on all projects - if nothing has to be built - is on the 7-core-i7-iMac-OSX/CI-VM about 1h at the moment, which is mainly due to the rsync access to Jenkins master...

A full rebuild of all projects (excluding qt5 and cmake) takes about 2,5 hrs (for 118 projects as of November 2014):

$ cd ~/scripts; time ./all-tiers-install.sh rebuild
...
real	143m12.696s
user	193m56.993s
sys	42m30.157s

while a great portion of that is also due to the deployment of all built binaries to Jenkins master.

Building including qt5 with a rebuild of all projects (except cmake) needs 3 hrs (for 118 projects as of November 2014):

$ cd ~/scripts; time (./install.sh qt5 rebuild && ./all-tiers-install.sh rebuild)
...
real	189m14.808s
user	328m31.106s
sys	64m10.909s

where the user time indicates that qt5 increased the parallel building tremendously.

With tests enabled building everything would run for about 6 hrs (for 118 projects as of November 2014).