Adding builder priority to Buildbot 0.7.10

I’m currently managing the automated build/test system for my project (the most recent of many hats that I’ve been required to wear during my tenure). We are using Buildbot to manage the builds for three platforms (Windows, Linux, Mac OS X), two architectures (32-bit and 64-bit), and four branches of development (one new product, two branches of a legacy product, and N-number of growing private developer branches). Doing the numbers, and trying to apply the result to six available machines for performing automated builds and test cases, it becomes clear that some level of priority management is needed to ensure the higher-priority builds complete before the lower-priority are serviced.

(Preface: This article discusses the usage of an open-source build automation tool known as Buildbot. Some familiarity with the tool — better, some need for builder prioritization within buildbot — would be helpful.)

Buildbot 0.7.10 includes prioritizations based upon the submission time of a build request (Buildbot 0.7.9, the version currently in use on my Buildbot master, doesn’t even provide this). Unfortunately, this level of prioritizing is insufficient to my needs. I need to prioritize the builders themselves, not just the build requests across all builders, so that certain builds will be serviced on the same build slave prior to others. For this reason, I’ve made some slight modifications to the Buildbot 0.7.10 sources in order to add this capability. Herein, I document the changes I’ve made in case you need this ability as well (or care to suggest improvements).

The Master Config

In my Buildbot master.cfg file, I have added a new dictionary item called ‘priority’ to each builder definition.  Priorities of zero (0) have the highest serviceability, with all other positive integers being inversely proportional to their priority (i.e., the larger the number, the increasingly lower its priority).  For example, my builder definitions now look something like this:

b1 = {'name': "TOP_Win32",
      'slavename': "Pyramid_8Core",
      'builddir': "win_32",
      'factory': f1,
      'priority': 0,
     }
b_bob_win = {'name': "TOP_Win64_bob",
             'slavename': "Pyramid_8Core",
             'builddir': "win_64_bob",
             'factory': f_bob_win,
             'priority': 3,
            }

Buildbot Modifications

To accommodate this approach, several small changes need to be made to a number of the Buildbot 0.7.10 files. You’ll need to find these files within your Buildbot 0.7.10 installation. For example, given Python 2.4 under Linux, you might find these files in:

/usr/lib/python2.4/site-packages/buildbot-0.7.10/buildbot

All file references made below will be relative to your active Buildbot installation location.

[ process/builder.py ]

We modify the Builder class in this file to contain priorities set in the builder definition of the master.cfg file. The __init__() method receives this new code (placed anywhere within):

self.priority = 0
if setup.has_key('priority'):
    self.priority = int(setup['priority'])

Further down, in the compareToSetup() method, code is added to deal with changes in the priority setting (for example, when executing “buildbot reconfig master”):

if setup.has_key('priority'):
    if int(setup['priority']) != self.priority:
        diffs.append('priority changed from %d to %d' \
                     % (self.priority, int(setup['priority'])))

The Builder class also requires this simple new method, used in sorting builders by priority:

def getPriority(self):
    return self.priority

[ master.py ]

This file will contain the largest modifications of the two. We are going to enhance the sorting function found in the maybeStartAllBuilds() method. Following the definition of the _sort() function within that method, remove these trailing lines:

builders.sort(cmp=_sortfunc)
for b in builders:
    b.maybeStartBuild()

and replace them with the following code:

builders.sort(key=lambda b: b.getPriority())

priority_high = builders[0].getPriority()
priority_low = builders[-1].getPriority()

sorted_builders = []
while priority_high <= priority_low:
    priority_builders = [b for b in builders if b.getPriority() == priority_high]
    priority_builders.sort(cmp=_sortfunc)
    sorted_builders += priority_builders
    priority_high += 1

for b in sorted_builders:
    b.maybeStartBuild()

Briefly, the replacement code does the following:

  • First, the local builders[] list is sorted into priority order, with the highest priority builders at the front, and the lowest priority builders bringing up the rear.
  • Next, the priority boundaries are gathered into local variables priority_high and priority_low, and these priorities are used to iterate over the builders[] list.
  • Each priority iteration sorts the specific priorities based upon the submission times of their build requests (if any). When complete, the effect is that the builder list is now sorted on two key values, builder priority and build request submission time.
  • Finally, the resulting two-key list is iterated over, with the highest-priority builder having the oldest build request at the top of the list, and the lowest-priority builder with the newest build request at the bottom. It should be noted that, regardless of whether or not a builder contains pending build requests, the maybeStartBuild() method needs to be called. This method updates the status display, and failing to call it can leave the Web interface displaying inaccurate information (e.g., the Waterfall can display a “building” status for an “idle” slave).

Wrapping Up

These changes do not go so far as to “abort” active builds based upon priority — in other words, killing builds in progress so that higher-priority builds can use the slave. Such actions can get messy if they are not handled correctly (e.g., the aborted build needs to be re-queued with the same submission time). These changes I’ve provided here only affect “pending” builds, making sure priorities give the next available slave to the more-critical builder with waiting build requests. If a “pending” priority queuing mechanism is too tame for your needs, it shouldn’t take much to modify these changes to make an “abort” priority queue.

So far, these modifications to Buildbot 0.7.10 appear to be working well for my specific situation. They may work as well for you, or you may need something slightly different. In any case, I hope these changes can serve as a starting point for achieving your specific needs. Good luck. 🙂

Advertisements

5 thoughts on “Adding builder priority to Buildbot 0.7.10

  1. A fascinating discussion is definitely worth comment.
    I do believe that you ought to write more about this issue, it might not
    be a taboo subject but generally folks don’t talk about
    these issues. To the next! Best wishes!!

  2. Wow! This blog looks just like my old one! It’s on a entirely different subject but it has pretty much the same page layout and design.
    Outstanding choice of colors!

Comments are closed.