Current status of setting up the CI system

This page shall describe what has to be done to get a KDE/CI system up and running on OSX. Partially this is achieved by using a few ports from MacPorts.

Start with a fresh Mavericks/MacPorts install

Currently using the official MacPorts version 2.3.1 is recommended. To be on the safe side make sure your disk has about 60G space in total.

However, currently a Mavericks 10.9.4 KDE/CI system on an i7-iMac uses something like this:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/disk0s2      64G   40G   25G  62% /

that is after all frameworks have been built and installed as shown further down.

Getting a MacPorts-compatible bash environment

Install ports for MacPorts' bash

$ sudo port install bash bash-completion

and set up the ​bash environment for MacPorts use regarding search paths (PATH, MANPATH)

Apple developer tools

Install

• Xcode from AppStore (and agree actively to the developer license agreement)
• developer command line tools

  $ xcode-select --install
  

Tools already present on OSX

Some tools are already installed on OSX due to Xcode (like git, svn, ssh, rsync, make, clang etc.) they don't need to be installed via MacPorts for now. Should a more up-to-date version of any one of these be necessary for the CI system it could be added later.

Some more software supplied by MacPorts

Install ports needed for the CI scripts as well as KF5:

$ # Ports for CI:
$ sudo port install gnutar py27-lxml bazaar mercurial wget automake autoconf cppcheck lcov
$
$ # Ports for KF5:
$ sudo port install shared-mime-info docbook-xml docbook-xsl giflib boost

Basic setup of CI system

In order to start building up a build slave for KDE's Jenkins-based CI system it is in principle needed to clone ​websites/build-kde-org, BUT further down it will become clear, that it is a good idea to track all changes to be made for websites/build-kde-org in a dedicated branch. That is why the ​dedicated repository clone "mp-osx-ci" has been introduced which has the branch 'mp-osx-ci' containing all changes needed for a CI system on OSX.

Create some folders, clone and update CI system (assuming a user "kdeci" with its home directory "/Users/kdeci"):

$ cd
$ mkdir -p WC/KDECI-builds;
$ cd WC
$ git clone git://anongit.kde.org/clones/websites/build-kde-org/kaning/mp-osx-ci.git
$ cd build-kde-org
$ git checkout mp-osx-ci
$ ./update-setup.sh                    # Ignore 404 occurring due to error cloning ECMA262 by hg

The update shell script created a folder ~/scripts in which all subsequent commands will have to be executed.

(BTW, if you were checking out the original repository from ​git://anongit.kde.org/websites/build-kde-org you'd have to check out the "production" branch.)

Note: All configuration settings listed below should already be present in the configuration set up in branch 'mp-osx-ci'.

1st step: Qt5 build

Prepare environment for Qt5 build

Install additional ports specific for Qt5 as well as KF5, set correct path to KDE-install directory in config/build/darwin-mavericks.cfg (note, some of the to-be-installed ports got already installed further up as dependencies for other tools, they're listed here to make the dependency on them explicit):

$ sudo port install zlib openssl dbus jpeg tiff libmng libpng mysql55 pkgconfig sqlite2 icu pcre libiconv
$ sudo port uninstall cmake            # Remove MacPorts' cmake (which came into the system as build-dependency for mysql55)
$ cd ~/scripts
$ cat config/build/darwin-mavericks.cfg
[DEFAULT]
scriptsLocation=/Users/kdeci/scripts
opSys=darwin
architecture=mavericks
compiler=clang

[General]
installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/
$ python2.7 tools/prepare-environment.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/qt5
...
$

BTW, it could be important to also install freetype for Qt5, but it hasn't been clarified yet whether it should be there or not.

Set up build environment for Qt5

It is required to build the debug version. Also make sure that all libs can be found by adding search paths, use "-dbus-linked" to make the ​session bus work and avoid inclusion of MacPorts' glib2:

$ cd ~/scripts; cat config/build/qt5/darwin-mavericks.cfg
[Build]
configureCommand=%(configureExecutable)s -debug -separate-debug-info -system-zlib -system-libpng -system-libjpeg -system-sqlite -dbus-linked -plugin-sql-mysql -nomake examples -confirm-license -opensource -prefix {instPrefix} -no-framework -v -I/opt/local/include -L/opt/local/lib -no-glib

Include MySQL's binary path into PATH e.g. in .macports/profile (assuming /opt/local as MacPort's prefix):

export PATH=/opt/local/lib/mysql55/bin:$PATH

Set up SSH for rsync access to build host, then start building Qt5:

$ cat ~/.ssh/config 
Host build.kde.org
   Port 2022
$ ssh-keygen -C YOUR@EMAIL.ADDRESS       # <-- send your public SSH key to Ben Cooksley in order to get access to the build server
$ python2.7 tools/perform-build.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/qt5
...

A very important issue for Qt5 and KF5 is that QStandardPaths gets set appropriately for OSX in general and especially for the OSX/CI system, as this installs all frameworks and apps into separate install roots. To enable Qt5/KF5 apps to find the correct locations for files (for configuration, read-only data, writeable data, etc.) it is necessary to patch QStandardPaths. This is done as a temporary hack in

~/scripts/patches/qt5/kf5-qt5/patch-qstandardpaths_mac.cpp.diff

and still required more work. The patch enables kdoctools and kdelibs4support to locate their files successfully (which had up to now to be handled using an even nastier workaround), but apart from that there are still problems with config files. In short: this is still work in progress...

Helper bash scripts

KDE's Jenkins-based CI system mainly makes use of these two python scripts:

tools/prepare-environment.py
tools/perform-build.py

For qt5 they've above been used in order to show the workflow. But in daily use it is more practical to consider some helper bash scripts which ease all actions needed to run project builds on an OSX/CI system for which there's not yet the Jenkins integration available.

The various projects on the CI system have to be handled by replacing "PROJECT" with the project name of the corresponding framework or application:

$ python2.7 tools/prepare-environment.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/PROJECT
...
$ (cd ~/WC/KDECI-builds/PROJECT; git checkout jenkins)
...
$ python2.7 tools/perform-build.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/PROJECT
...

or alternatively by using ​helper bash scripts in ~/scripts/ present in the mp-osx-ci branch of the mp-osx-ci repository

$ ./prepare.sh PROJECT
...
$ ./build.sh PROJECT
...

where prepare.sh takes care of creating the build directory if it is not yet existing as well as checks out the branch 'jenkins'. Alternatively one can combine these steps using:

$ ./install.sh PROJECT [rebuild]

The 'rebuild' option can be used to trigger calling build.sh even if prepare.sh didn't signal any changes in the remote project repository.

Installation of cmake

Cmake needs a manual checkout of its sources (which is demonstrated here the explicit way again):

$ python2.7 tools/prepare-environment.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/cmake
...
$ git clone git://cmake.org/cmake.git ~/WC/KDECI-builds/cmake
$ python2.7 tools/perform-build.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/cmake
...
$ 
$ # Alternatively you can simply use this script (which does all of the above for you):
$ ./install.sh cmake

Important note: Currently it seems to be necessary to NOT to install the latest cmake version on OSX, as this causes ​build problems for framework kcoreaddons later on. The commit 3c0d87645f5861e9fd6fd78855697f77ef77144e was recommended by Cristian Onet. This is done secretly in prepare.sh already.

Setting up the build environment

The branch 'mp-osx-ci' contains all required configuration settings for OSX. The most important shall be pointed out here:

• For KF5 on OSX one has to modify the configuration settings by commenting out the kauth dependency on polkit-qt-1, since that dependency is only needed on Linux.

• Problems appeared for some frameworks since kconfig_compiler_kf5.app doesn't yet get installed on OSX (the build scripts error out with "/Applications/KDE/kconfig_compiler_kf5.app/Contents/MacOS/kconfig_compiler_kf5 as location is wrong"). This can be fixed for now introducing a temporary configuration file for kconfig which sets CMAKE_INSTALL_BUNDLEDIR to make sure that the application bundle gets installed below PREFIX/lib/libexec/.

• CMAKE_INSTALL_BUNDLEDIR and DATA_INSTALL_DIR need to be set for OSX-like directories.

• Disable searching for X11 by setting CMAKE_DISABLE_FIND_PACKAGE_X11=ON.

• Disable running tests for now, until dbus works as expected and test applications can be started normally.

• DYLD_LIBRARY_PATH has only temporally been added due to wrong linking of phonon (e.g. for kate and kwrite).

• Disable coverage for plasma-framework for now until a coverage tool for clang is integrated.

Installation of other projects and KDE frameworks

Specific frameworks can be installed using install.sh , but one can also build all frameworks of a specific tier as a whole using

$ # This builds all frameworks of tier 1
$ ./tier-install.sh 1
...
$ # This builds all frameworks of tier 2, but forces rebuilding every project regardless of
$ # whether git pulled in changes or not (useful when e.g. qt5 or cmake was updated)
$ ./tier-install.sh 2 rebuild
...
$ # This rebuilds all frameworks of tier 3 starting at project kio (i.e. skipping all predecessor frameworks)
$ ./tier-install.sh 3 kio rebuild
...

All frameworks of all tiers can be build easily using

$ ./all-tiers-install.sh [rebuild]

which will iterate trough tiers 0 up to 5, with tiers 0 and 5 being just "virtual" tiers - which were introduced to more easily build some projects.

KF5 builds have been successfully done so far for these projects:

Additional dependencies (referred to as "Tier 0"):
 - extra-cmake-modules
 - phonon

Tier 1 frameworks (COMPLETE):
 - attica
 - kapidox
 - karchive
 - kcodecs
 - kconfig
 - kcoreaddons
 - kdbusaddons
 - kdnssd
 - { kglobalaccel }
 - kguiaddons
 - ki18n
 - kidletime
 - kimageformats
 - kitemmodels
 - kitemviews
 - kplotting
 - kwidgetsaddons
 - { kwindowsystem }
 - solid
 - sonnet
 - threadweaver

Tier 2 frameworks (COMPLETE):
 - kauth
 - kcompletion
 - { kcrash }
 - kdoctools
 - kjobwidgets
 - { kpty }
 - kunitconversion

Tier 3 frameworks incl. phonon (ALMOST COMPLETE):
   [ATTENTION: the build order matters here]
 - kconfigwidgets
 - kservice
 - kiconthemes
 - ktextwidgets
 - kxmlgui
 - kcmutils
 - kbookmarks
 - { kdesu }
 - kemoticons
 - knotifications
 - kwallet
 - kio
 - kdeclarative
 - kactivities
 - kparts
 - kdewebkit
 - kinit
 - kmediaplayer
 - knewstuff
 - knotifyconfig
 - ktexteditor
 - kdesignerplugin
 - kded
 - plasma-framework

Tier 4:
 - frameworkintegration

Porting aids:
 - kjs
 - kdelibs4support
 - kjsembed
 - khtml
 - kross
 - krunner

 Meta-project including tiers 1-4 + porting aids:
 - kf5umbrella

Curly brackets in the above lists mark frameworks marked on ​KDE's official API page as having problems on MacOSX, since plasma won't be running for now.

Having the very compact ​Gentoo's Project:KDE/Frameworks wiki page around turned out to be very helpful while building the above KDE frameworks manually. More info is found on ​KDE's official API page and the above lists have been aligned with it: nice dependency graphs for every framework can be found there by clicking the "Dependencies" link on each frameworks page, some more (overview) graphs are ​here.

As of July 5th 2014 all KF5 frameworks build successfully.

Troubleshooting for projects' dependencies

​Dependency data for the CI system is being downloaded by ~/scripts/update-setup.sh to ~/scripts/dependencies/. This folder contains files defining the dependencies (see subsections below) as well as some tools.

Sometimes it might be needed to add a new entry to the KF5-specific (logical) dependency data in case a project cannot find a required framework, although it is already available on the CI system! Details concerning this can be found on a dedicated page.

Notes concerning some files installed by the frameworks

All notes regarding the placement of certain files (apps, config, data, etc.) have been moved aside to a dedicated page as the patching of Qt5's QStandardPaths is still work in progress.

KF5 software beyond the basic frameworks

Since the basic set of frameworks is now ready for use on OSX one can go ahead and start building other software depending on KF5. A first step is building kdesupport-svn which is a prerequisite for further projects.

Important projects are e.g. KDevelop, KMyMoney, but also ​KDE-EDU applications on their ​route to KF5 can serve as good test cases. This is referred to as "Tier 5":

Done:
 - kdesupport-svn
 - kio-extras
 - kde-cli-tools
 - khelpcenter (%)
 - kate (+)
 - libkomparediff2
 - kfilemetadata
 - kdevplatform
 - kinfocenter (%)
 - kdevelop (.)
 - okteta  (o)
 - qca
 - konversation (%)
 - konsole ($)
 - akonadi
 - alkimia (needs port gmp installed)
 - prison (needs ports qrencode and "libdmtx -x11" [which pulls also these in: ImageMagick djvulibre fftw-3 fontconfig freetype ghostscript jbig2dec lcms2 urw-fonts webp]) 
 - kmymoney (=1)
 - kde-baseapps (2?)

KDE edu:
 - libkdeedu (§)
 - libkeduvocdocument
 - analitza
 - kalgebra (!)
 - step (.)
 - kgeography (%)
 - kig (*)
 - kanagram (.)
 - parley (x)
 - kstars (*, needs port eigen3 installed)
 - kqtquickcharts
 - marble (?)

Planned:
 - artikulate (3)
 - kiten (0)
 - kdepimlibs (4)
 - oxygen (5)
 - pairs (0)
 - rocs (0)
 - klettres (0)

Unneeded on OSX (tried just for the fun of it):
 - kwin (a)
 - libkscreen (a)
 - kde-runtime (a,b)
 - ktp-common-internals (c)
 - libksysguard (d)
 - khotkeys (e)
 - kmenuedit (e)
 - milou (f)
 - powerdevil (f)
 - baloo (g)
 - xapian

Legend:

NOTE 1: Many applications throw QDBusError("org.freedesktop.DBus.Error.ServiceUnknown", ... ) errors which might be due to the lack of proper search paths for Qt. This is WIP.

NOTE 2: Currently DYLD_LIBRARY_PATH is globally set for phonon.

(%) starts, but unsuccessfully tries to fire up DrKonqi, then keeps running but is not showing sensible content, main menu missing
(+) application has to be started with "QT_GRAPHICSSYSTEM=raster" (​b.k.o. issue 337140)
(.) exits because of an ASSERT
(o) throws QDBusError("org.freedesktop.DBus.Error.ServiceUnknown", ... )
($) does not start because it can't find dbus atm, i.e. exits gracefully
(=) crashes at startup although with and without setting raster graphics
(§) according to PovAddictW not very much useful in there anymore these days (kalgebra doesn't need it)
(!) application can actually be executed successfully
(*) crashes because some config files can't be found
(x) starts, but some config files can't be found
(?) starting application still untested

(0) not ported yet
(1) kmymoney has some ​(temporally) optional frameworks
(2) needs fixing spaces in DTD path (see also ​patch regarding docbook DTD)
(3) Qt5GStreamer needed
(4) build failure due to ​kdepimlibs/akonadi
(5) needs kwin, but according to ​info for gentoo this can be optional

(a) FWs require Qt5X11Extras, although it is switched off in global configuration using -DCMAKE_DISABLE_FIND_PACKAGE_X11=ON
(b) kde-runtime is not needed, since its functionality is currently getting ported into other KF5 FWs step by step
(c) ktp-common-internals requires TelepathyQt5
(d) libksysguard is ​broken (see also ​this)
(e) need libksysguard
(f) needs baloo
(g) needs ​xapian