buildbot improvements

[mojca (Mojca Miklavec)]

I'm opening a single ticket to list and brainstorm ideas for further improvements of the new buildbot setup. Please note that each individual item listed here would deserve its own ticket, but it makes no sense to open so many tickets at this early stage.

• provide means to build just a single port without any subports (for example, allow building just wxWidgets-3.0 without wxgtk-3.0 and others)
• on top of the portlist (portname) variable, introduce a new variable that holds variants (say +quartz -x11) and allow building the port with those variants (the idea is not to come up with a complete list of useful builds, but to set the necessary building blocks to allow building quartz variants of ports or maybe other tricks that might occasionally be needed)
• triggering the individual builds on the builder is currently in random order (not sure if this is possible to fix)

Display / UI

• create a hyperlink from the builder to the watcher (currently there's just url, but it's not a clickable hyperlink)
• create hypherlinks to individual builds from the console page (there's currently just a hyperlink that points to an useless # on the same page)
• create navigation button for individual builds (back and forward)
• disable tracking of remaining build time based on statistics (unless the statistics gets considerably better and actually tracks builds of individual ports)
• Is it possible to display URLs (on the watcher) to triggered builds as soon as the builds start (rather than waiting until all builds are finished)?
• (not sure if doable) pending forced build requests don't display any info at all until they are started (one doesn't know which port builds were forced)
• it would be nice if hyperlinks from the watcher to the builders would also contain the port name being built
• (the port builder could theoretically display a new property with all ports being built by the same watcher; no added benefit, just giving extra info to anyone browsing the logs; super low priority, if desired at all)
• it would be handy if the SVN revision (when applicable) was also shown on the port builder (with all the sugars ... hyperlinks, commit message etc.), not just the port watcher

Log files

• provide a link to main.log
• provide a link to config.log if one exists (we can worry about more advanced situations with multiple logs later)
• remove stdio from the first step in builder with the sole purpose of displaying port name
• the builder should provide a list of dependencies to be installed as a separate log file, for example in the following format:

  Installing dependencies:
   - expat
   - jpeg
   - libiconv
   - zlib
   - libpng
   - ncurses
   - gettext
   - xz
   - tiff
  
  Installing dependency 1 of 9: expat ... [OK]
  Installing dependency 2 of 9: jpeg ... [OK]
  Installing dependency 3 of 9: libiconv ... [OK]
  Installing dependency 4 of 9: libpng ... [OK]
  Installing dependency 5 of 9: ncurses ... [OK]
  Installing dependency 6 of 9: gettext ... [FAIL]
  
  Error: installation of gettext failed.
  (optionally print the last few lines from `main.log`)
  

• provide a shorter log about the build progress, the equivalent of running port install without verbose/debug mode; we could that that either by filtering the logs and displaying just the relevant lines or by running the installation without debug more and get debug information from the main.log instead; needs some thinking
• steps/svn/logs/stdio on the watcher now contains email addresses (that's slightly bad for web spiders)
• the watcher should provide a summary of successful / broken triggered builds similar to the old status, for example something like:

  Building 1 of 20 foo ... [OK]
  Building 2 of 20 foo-devel ... broken dependency baz [FAIL]
  Building 3 of 20 bar ... broken dependency bar-2.0 [FAIL]
  Building - of 20 bar-1.0 ... obsolete [SKIP]
  Building 4 of 20 bar-2.0 ... failed before, no attempt to rebuild [FAIL]
  Building 5 of 20 bar-3.0 ... [OK]
  ...
  

Sending e-mails

• portwatcher should send notification about broken builds to all maintainers of the broken ports + the committer (or the user who forced the build)
• include the aggregated statistics in the email
  • which ports failed to build
  • whether they failed because of broken dependencies (which one) or due to a build error in port itself (maybe even the phase where the error happened), see the log file proposal above
• include the URL to the build (both the watcher and the builder if possible)

Improvements in the core

• if the port builds locally (rather that being fetched from the binary archive), it would be nice to store the build time in the database (with some caution when builds are broken)
  • I would like to run a command that would print the list of installed port sorted by the build time (say, to determine which ports are the most demanding/important to build on the buildbot)
  • this could be useful input for determining the remaining build time on the builder (if we ever figure out how to hack the buildbot to read that piece of information from an external source)

Statistics

• create a simple web site that collects statistics in a table, containing the following variables:
  • port name
  • version
  • revision
  • explicit variants (usually none, but +universal could be one)
  • all variants
  • OS version
  • architecture
  • builder name
  • number or link to the watcher
  • number or link to the builder
  • success: OK | FAIL | (FAIL-skip)
  • uploaded: True | False
• and then prints out the list of latest builds grouped by Portfile + URLs to binary archives

Organisation

• move the old buildbot to a place of historic interest and rename buildbot-test to buildbot
• potentially move more logic of master.cfg to a perl module
• move the list of builders from hardcoded python code to a configuration file