Current status of setting up the CI system

This page shall describe what has to be done to get a KDE/CI system up and running on OSX. Partially this is achieved by using a few ports from MacPorts.

Start with a fresh Mavericks/MacPorts install

Currently using the official MacPorts version 2.3.1 is recommended. To be on the safe side make sure your disk has about 60G space in total.

However, currently a Mavericks 10.9.4 KDE/CI system on an i7-iMac uses something like this:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/disk0s2      64G   40G   25G  62% /

that is after all frameworks have been built and installed as shown further down.

Getting a MacPorts-compatible bash environment

Install ports for MacPorts' bash

$ sudo port install bash bash-completion

and set up the ​bash environment for MacPorts use regarding search paths (PATH, MANPATH)

Apple developer tools

Install

• Xcode from AppStore (and agree actively to the developer license agreement)
• developer command line tools

  $ xcode-select --install
  

Tools already present on OSX

Some tools are already installed on OSX due to Xcode (like git, svn, ssh, rsync, make, clang etc.) they don't need to be installed via MacPorts for now. Should a more up-to-date version of any one of these be necessary for the CI system it could be added later.

Some more software supplied by MacPorts

Install ports needed for the CI scripts as well as KF5:

$ sudo port install gnutar py27-lxml bazaar mercurial wget automake autoconf cppcheck shared-mime-info docbook-xml docbook-xsl giflib boost

Basic setup of CI system

Create some folders, clone and update CI system (assuming a user "kdeci" with its home directory "/Users/kdeci"):

$ cd
$ mkdir -p WC/KDECI-build;
$ cd WC
$ git clone git://anongit.kde.org/websites/build-kde-org
$ cd build-kde-org
$ git checkout production
$ ./update-setup.sh                    # Ignore 404 occurring due to error cloning ECMA262 by hg

The update shell script created a folder ~/scripts in which subsequent commands will have to be executed.

1st step: Qt5 build

Prepare environment for Qt5 build

Install additional ports specific for Qt5 as well as KF5, set correct path to KDE-install directory in config/build/darwin-mavericks.cfg:

$ sudo port install zlib openssl dbus jpeg tiff libmng libpng mysql55 pkgconfig sqlite2
$ sudo port uninstall cmake            # Remove MacPorts' cmake (which came into the system as build-dependency for mysql55)
$ cd ~/scripts
$ cat config/build/darwin-mavericks.cfg
[DEFAULT]
scriptsLocation=/Users/kdeci/scripts
opSys=darwin
architecture=mavericks
compiler=clang

[General]
installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/
$ python2.7 tools/prepare-environment.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/qt5
...
$

Set up build environment for Qt5

Make sure all libs can be found by adding search paths and avoid inclusion of MacPorts' glib2:

$ cd ~/scripts; cat config/build/qt5/darwin-mavericks.cfg
[Build]
configureCommand=%(configureExecutable)s -release -system-zlib -system-libpng -system-libjpeg -system-sqlite -dbus -plugin-sql-mysql -nomake examples -confirm-license -opensource -prefix {instPrefix} -no-framework -v -I/opt/local/include -L/opt/local/lib -no-glib

Consider here the use of options "-debug -separate-debug-info" instead of "-release".

Include MySQL's binary path into PATH e.g. in .macports/profile (assuming /opt/local as MacPort's prefix):

export PATH=/opt/local/lib/mysql55/bin:$PATH

Set up SSH for rsync access to build host, then start building Qt5:

$ cat ~/.ssh/config 
Host build.kde.org
   Port 2022
$ ssh-keygen -C YOUR@EMAIL.ADDRESS       # <-- send your public SSH key to Ben Cooksley in order to get access to the build server
$ python2.7 tools/perform-build.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/qt5
...

Installation of cmake

Cmake needs a manual checkout of its sources:

$ python2.7 tools/prepare-environment.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/cmake
...
$ git clone git://cmake.org/cmake.git ~/WC/KDECI-build/cmake
$ python2.7 tools/perform-build.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/cmake
...

Keeping build environment up-to-date

For KF5 one has to modify the configuration settings by commenting out the kauth dependency on polkit-qt-1, since that dependency is only needed on Linux:

$ cd ~/scripts
$ ./update-setup.sh
$ vi config/base/kf5-qt5 
$ grep kauth config/base/kf5-qt5
#frameworks/kauth: kdesupport/polkit-qt-1
$

Problems appeared for some frameworks since kconfig_compiler_kf5.app doesn't yet get installed on OSX (the build scripts error out with "/Applications/KDE/kconfig_compiler_kf5.app/Contents/MacOS/kconfig_compiler_kf5 as location is wrong"). This can be fixed for now introducing a temporary configuration file for kconfig:

$ cat ~/scripts/config/build/kconfig/darwin-mavericks.cfg 
[DEFAULT]
configureExtraArgs=-DCMAKE_INSTALL_BUNDLEDIR="lib/libexec/kf5"

which makes sure that the application bundle gets installed below PREFIX/lib/libexec/.

In order to always have the latest KDE/CI running it makes sense to temporally stash away the changes made locally and upgrade via git

$ # Check what has changed locally:
$ git diff
diff --git a/config/base/kf5-qt5 b/config/base/kf5-qt5
index 16abdf6..fff9647 100644
--- a/config/base/kf5-qt5
+++ b/config/base/kf5-qt5
@@ -18,7 +18,7 @@ general/vc: -qt5
 general/shared-desktop-ontologies: -qt5
 
 # KDE Frameworks
-frameworks/kauth: kdesupport/polkit-qt-1
+#frameworks/kauth: kdesupport/polkit-qt-1
 kde/*: general/kdesupport-svn
 kde/workspace/plasma-workspace: general/libdbusmenu-qt
 
diff --git a/config/build/darwin-mavericks.cfg b/config/build/darwin-mavericks.cfg
index cb62c49..0648464 100644
--- a/config/build/darwin-mavericks.cfg
+++ b/config/build/darwin-mavericks.cfg
@@ -1,7 +1,9 @@
 [DEFAULT]
+scriptsLocation=/Users/kdeci/scripts
 opSys=darwin
 architecture=mavericks
 compiler=clang
+configurePlatformArgs=-DCMAKE_INSTALL_BUNDLEDIR="Applications/KF5" -DDATA_INSTALL_DIR="Library/Application Support" -DCMAKE_DISABLE_FIND_PACKAGE_X11=ON
 
 [General]
-installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/
\ No newline at end of file
+installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/
diff --git a/tools/environment-generator.py b/tools/environment-generator.py
index 37dcbae..e46a13c 100644
--- a/tools/environment-generator.py
+++ b/tools/environment-generator.py
@@ -31,7 +31,7 @@ environment = manager.generate_environment(True)
 neededVariables = [
        'CMAKE_PREFIX_PATH', 'KDEDIRS', 'PATH', 'LD_LIBRARY_PATH', 'PKG_CONFIG_PATH', 'PYTHONPATH',
        'PERL5LIB', 'QT_PLUGIN_PATH', 'QML_IMPORT_PATH', 'QML2_IMPORT_PATH', 'XDG_DATA_DIRS',
-       'XDG_CONFIG_DIRS', 'QMAKEFEATURES', 'XDG_CURRENT_DESKTOP'
+       'XDG_CONFIG_DIRS', 'QMAKEFEATURES', 'XDG_CURRENT_DESKTOP', 'DYLD_LIBRARY_PATH'
 ]
 
 # Generate the shell format environment file, suitable for sourcing
$ 
$ git status
On branch production
Your branch is up-to-date with 'origin/production'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   config/base/kf5-qt5
	modified:   config/build/darwin-mavericks.cfg

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	build.sh
	dependencies/
	kapidox/
	kde_projects.xml
	poppler-test-data/
	prepare.sh

no changes added to commit (use "git add" and/or "git commit -a")
$
$ # Updating CI scripts
$ cd ~/scripts; git stash; git pull; git stash pop

Note 1: It may be that the DATA_INSTALL_DIR setting is actually not needed!

Note 2: DYLD_LIBRARY_PATH has only temporally been added due to wrong linking of phonon (e.g. for kate and kwrite).

Installation of other projects and KDE frameworks

The various frameworks have to be handled by replacing "PROJECT" with the framework's project name:

$ python2.7 tools/prepare-environment.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/PROJECT
...
$ (cd ~/WC/KDECI-build/PROJECT; git checkout jenkins)
...
$ python2.7 tools/perform-build.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-build/PROJECT
...

or alternatively by using ​scripts from our MacPorts/KDE git repository

$ ./prepare.sh PROJECT
...
$ ./build.sh PROJECT
...

where prepare.sh takes care of creating the build directory if it is not yet existing as well as checks out the branch 'jenkins'. This is only needed when building a project manually, otherwise it gets one by the jenkins slave. Alternatively one can combine both of these steps using:

$ ./install.sh PROJECT

All frameworks of a specific tier can be build as a whole using

$ # This builds all frameworks of tier 1
$ ./tier-install.sh 1
...
$ # This builds all frameworks of tier 2, but forces rebuilding every project regardless of
$ # whether git pulled in changes or not (useful when e.g. qt5 or cmake was updated)
$ ./tier-install.sh 2 rebuild
...

In order to be able to build KF5 frameworks of tier 3 it is - as a temporary workaround - necessary to copy kdoctools' files from its install directory to where framework builds can find them (i.e. below "/Library/Application Support"):

$ sudo rm -rf /Library/Application\ Support/kf5/kdoctools/ ; sudo cp -Rp /opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kdoctools/inst/Library/Application\ Support/kf5 /Library/Application\ Support

{It should be possible to avoid this by patching Qt5's QStandardPaths in an appropriate manner, which is atm still investigated...}

KF5 builds have been successfully done so far for:

Additional dependencies:
 - extra-cmake-modules
 - phonon

Tier1 frameworks (COMPLETE):
 - attica
 - kapidox
 - karchive
 - kcodecs
 - kconfig
 - kcoreaddons
 - kdbusaddons
 - kdnssd
 - { kglobalaccel }
 - kguiaddons
 - ki18n
 - kidletime
 - kimageformats
 - kitemmodels
 - kitemviews
 - kplotting
 - kwidgetsaddons
 - { kwindowsystem }
 - solid
 - sonnet
 - threadweaver

Tier2 frameworks (COMPLETE):
 - kauth
 - kcompletion
 - { kcrash }
 - kdoctools
 - kjobwidgets
 - { kpty }
 - kunitconversion

Tier3 frameworks incl. phonon (ALMOST COMPLETE):
   [ATTENTION: (1) above workaround needed, (2) the build order matters here]
 - kconfigwidgets
 - kservice
 - kiconthemes
 - ktextwidgets
 - kxmlgui
 - kcmutils
 - kbookmarks
 - { kdesu }
 - kemoticons
 - knotifications
 - kwallet
 - kio
 - kactivities
 - kdeclarative
 - kparts
 - kdewebkit
 - kinit
 - kmediaplayer
 - knewstuff
 - knotifyconfig
 - ktexteditor
 - kdesignerplugin
 - kded
 - plasma-framework

Tier 4:
 - frameworkintegration

Porting aids:
 - kjs
 - kdelibs4support
 - kjsembed
 - khtml
 - kross
 - krunner

 Meta-project including tiers 1-4 + porting aids:
 - kf5umbrella

Curly brackets in the above lists mark frameworks marked on ​KDE's official API page as having problems on MacOSX, since plasma won't be running for now.

Having the very compact ​Gentoo's Project:KDE/Frameworks wiki page around turned out to be very helpful while building the above KDE frameworks manually. More info is found on ​KDE's official API page and the above lists have been aligned with it: nice dependency graphs for every framework can be found there by clicking the "Dependencies" link on each frameworks page, some more (overview) graphs are ​here.

As of July 5th 2014 all KF5 frameworks build successfully. {NOTE: Still using the above workaround for the CI system!}

Notes concerning some files installed by the frameworks

Installs into "/Library/Application Support"

These folders should be copied into the MacPorts/KDE-CI system's "/Library/Application Support" directory (as it was done for as a temporary work-around for kdoctools above):

$ cd ~/WC/KDECI-builds; find . -type d -wholename "./*/local-inst/opt/kde/*/Library/Application Support/*" | egrep "/kf5$"
./frameworkintegration/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/frameworkintegration/inst/Library/Application Support/kf5
./kauth/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kauth/inst/Library/Application Support/kf5
./kconfigwidgets/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kconfigwidgets/inst/Library/Application Support/kf5
./kdelibs4support/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kdelibs4support/inst/Library/Application Support/kf5
./kdesignerplugin/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kdesignerplugin/inst/Library/Application Support/kf5
./kdoctools/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kdoctools/inst/Library/Application Support/kf5
./khtml/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/khtml/inst/Library/Application Support/kf5
./kio/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kio/inst/Library/Application Support/kf5
./kjs/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kjs/inst/Library/Application Support/kf5
./knewstuff/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/knewstuff/inst/Library/Application Support/kf5
./kwidgetsaddons/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kwidgetsaddons/inst/Library/Application Support/kf5
./kxmlgui/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kxmlgui/inst/Library/Application Support/kf5
./sonnet/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/sonnet/inst/Library/Application Support/kf5

Problematic installs into "/Library/Application Support"

The following folders do not reside below a kf5 sub-directory of "/Library/Application Support" which seems to indicate that there is a problem in the corresponding CMake files of khtml and ktexteditor frameworks:

$  cd ~/WC/KDECI-builds; find . -type d -wholename "./*/local-inst/opt/kde/*/Library/Application Support/*" | grep -v "/kf5/" | egrep -v "/kf5$"
./khtml/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/khtml/inst/Library/Application Support/khtml
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/commands
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/files
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/files/quickcoding
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/files/quickcoding/cpp
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/indentation
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/libraries
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/script/libraries/emmet
./ktexteditor/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/ktexteditor/inst/Library/Application Support/katepart5/syntax

Installs into "/Applications/KF5"

These are all application packages which might have to be copied into /Applications/KF5 on the KDE/CI system:

$ cd ~/WC/KDECI-builds; find . -type d -wholename "./*/local-inst/opt/kde/*/frameworks/*/Applications/KF5/*.app" 
./kactivities/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kactivities/inst/Applications/KF5/kactivitymanagerd.app
./kded/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kded/inst/Applications/KF5/kded5.app
./kdelibs4support/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kdelibs4support/inst/Applications/KF5/kdebugdialog5.app
./kinit/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kinit/inst/Applications/KF5/kdeinit5.app
./kjsembed/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kjsembed/inst/Applications/KF5/kjscmd5.app
./kross/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kross/inst/Applications/KF5/kf5kross.app
./kwallet/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kwallet/inst/Applications/KF5/kwalletd5.app

Installs NOT into "/Applications/KF5"

These are all application packages which MIGHT NOT have to be copied into /Applications/KF5 on the KDE/CI system:

$ cd ~/WC/KDECI-builds; find . -type d -wholename "./*/local-inst/opt/kde/*/frameworks/*.app" | grep -v "Applications/KF5"
./kauth/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kauth/inst/lib/libexec/kauth/kauth-policy-gen.app
./kconfig/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kconfig/inst/lib/libexec/kf5/kconfig_compiler_kf5.app
./kinit/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kinit/inst/lib/libexec/kf5/start_kdeinit.app
./kinit/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kinit/inst/lib/libexec/kf5/start_kdeinit_wrapper.app
./kio/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kio/inst/lib/libexec/kf5/kio_http_cache_cleaner.app
./kjsembed/local-inst/opt/kde/install/darwin/mavericks/clang/kf5-qt5/frameworks/kjsembed/inst/bin/kjsconsole.app

kdesupport-svn - a prerequisite for further projects

In order to be able to build some KDE applications it is necessary to install ​kdesupport-svn, which is not automatically handled by the prepare script. Therefore one needs to check out its sources using Subversion and install it like this:

$ mkdir ~/WC/KDECI-builds/kdesupport-svn
$ svn checkout svn://anonsvn.kde.org/home/kde/trunk/kdesupport ~/WC/KDECI-builds/kdesupport-svn
$ ./build.sh kdesupport-svn

KF5 software beyond the basic frameworks

Since the basic set of frameworks is now ready for use on OSX one can go ahead and start building those. An important projects are KDevelop, KMyMoney, but also ​KDE-EDU applications can serve as good test cases.

Done:
 - kde-cli-tools
 - khelpcenter (-)
 - kate (-)
 - libkomparediff2
 - kfilemetadata
 - kdevplatform
 - kinfocenter (+)
 - akonadi
 - alkimia (needs port gmp installed)
 - konversation
 - qca
 - kdevelop
 - okteta
 - konsole

KDE edu:
 - libkdeedu (§)
 - analitza
 - kalgebra
 - step
 - kgeography

Planned:
 - prison (needs ports qrencode and "libdmtx -x11" [which pulls also these in: ImageMagick djvulibre fftw-3 fontconfig freetype ghostscript jbig2dec lcms2 urw-fonts webp]) 
 - kdepimlibs (1)
 - kmymoney (2)
 - kde-baseapps

Unneeded on OSX (tried just for the fun of it):
 - libkscreen (a)
 - kde-runtime (a,b)
 - ktp-common-internals (c)
 - libksysguard (d)
 - khotkeys (e)
 - kmenuedit (e)
 - milou (f)
 - powerdevil (f)
 - baloo (g)
 - xapian

(-) crashes on startup (​b.k.o. issue 337140) and needs DYLD_LIBRARY_PATH set for phonon
(+) crashes on startup
(§) according to PovAddictW not very much useful in there anymore these days (kalgebra doesn't need it)

(1) needs prison
(2) kmymoney misses ​some frameworks

(a) FWs require Qt5X11Extras, although it is switched off in global configuration using -DCMAKE_DISABLE_FIND_PACKAGE_X11=ON
(b) kde-runtime is not needed, since its functionality is currently getting ported into other KF5 FWs step by step
(c) ktp-common-internals requires TelepathyQt5
(d) libksysguard is ​broken (see also ​this)
(e) need libksysguard
(f) needs baloo
(g) needs ​xapian