I’m currently managing the automated build/test system for my project (the most recent of many hats that I’ve been required to wear during my tenure). We are using Buildbot to manage the builds for three platforms (Windows, Linux, Mac OS X), two architectures (32-bit and 64-bit), and four branches of development (one new product, two branches of a legacy product, and N-number of growing private developer branches). Doing the numbers, and trying to apply the result to six available machines for performing automated builds and test cases, it becomes clear that some level of priority management is needed to ensure the higher-priority builds complete before the lower-priority are serviced.
(Preface: This article discusses the usage of an open-source build automation tool known as Buildbot. Some familiarity with the tool — better, some need for builder prioritization within buildbot — would be helpful.)
Buildbot 0.7.10 includes prioritizations based upon the submission time of a build request (Buildbot 0.7.9, the version currently in use on my Buildbot master, doesn’t even provide this). Unfortunately, this level of prioritizing is insufficient to my needs. I need to prioritize the builders themselves, not just the build requests across all builders, so that certain builds will be serviced on the same build slave prior to others. For this reason, I’ve made some slight modifications to the Buildbot 0.7.10 sources in order to add this capability. Herein, I document the changes I’ve made in case you need this ability as well (or care to suggest improvements).
The Master Config
In my Buildbot master.cfg file, I have added a new dictionary item called ‘priority’ to each builder definition. Priorities of zero (0) have the highest serviceability, with all other positive integers being inversely proportional to their priority (i.e., the larger the number, the increasingly lower its priority). For example, my builder definitions now look something like this:
To accommodate this approach, several small changes need to be made to a number of the Buildbot 0.7.10 files. You’ll need to find these files within your Buildbot 0.7.10 installation. For example, given Python 2.4 under Linux, you might find these files in:
All file references made below will be relative to your active Buildbot installation location.
[ process/builder.py ]
We modify the Builder class in this file to contain priorities set in the builder definition of the master.cfg file. The
__init__() method receives this new code (placed anywhere within):
Further down, in the
compareToSetup() method, code is added to deal with changes in the priority setting (for example, when executing “buildbot reconfig master”):
The Builder class also requires this simple new method, used in sorting builders by priority:
[ master.py ]
This file will contain the largest modifications of the two. We are going to enhance the sorting function found in the
maybeStartAllBuilds() method. Following the definition of the
_sort() function within that method, remove these trailing lines:
and replace them with the following code:
Briefly, the replacement code does the following:
- First, the local
builderslist is sorted into priority order, with the highest priority builders at the front, and the lowest priority builders bringing up the rear.
- Next, the priority boundaries are gathered into local variables
priority_low, and these priorities are used to iterate over the
- Each priority iteration sorts the specific priorities based upon the submission times of their build requests (if any). When complete, the effect is that the builder list is now sorted on two key values, builder priority and build request submission time.
- Finally, the resulting two-key list is iterated over, with the highest-priority builder having the oldest build request at the top of the list, and the lowest-priority builder with the newest build request at the bottom. It should be noted that, regardless of whether or not a builder contains pending build requests, the
maybeStartBuild()method needs to be called. This method updates the status display, and failing to call it can leave the Web interface displaying inaccurate information (e.g., the Waterfall can display a “building” status for an “idle” slave).
These changes do not go so far as to “abort” active builds based upon priority — in other words, killing builds in progress so that higher-priority builds can use the slave. Such actions can get messy if they are not handled correctly (e.g., the aborted build needs to be re-queued with the same submission time). These changes I’ve provided here only affect “pending” builds, making sure priorities give the next available slave to the more-critical builder with waiting build requests. If a “pending” priority queuing mechanism is too tame for your needs, it shouldn’t take much to modify these changes to make an “abort” priority queue.
So far, these modifications to Buildbot 0.7.10 appear to be working well for my specific situation. They may work as well for you, or you may need something slightly different. In any case, I hope these changes can serve as a starting point for achieving your specific needs. Good luck. 🙂