I've started working on migrating the Firefox nightly builds to use the same release automation system that we've been developing for the past year or so for maintenance releases (Firefox and Thunderbird 1.5.0x and 2.0.0.x). The reason this is important is that each nightly release (installer, update, etc.) is practice for the real thing, and we should be using the same tools and verification processes wherever possible (right now both Nightlies and Releases use Tinderbox client [version 1] for build and l10n repack, all other aspects of the release process are not tested until the first release candidate. Well, we have a staging server that tests the release automation in isolation, but it's not the same as having real nightly testers looking at the results :) ).

The scope of the current release automation framework (Bootstrap) has been to leave as much of our existing process in place as possible, and not try to simplify or optimize. This kind of low-risk approach is the right thing to do when you're overhauling the release process on a maintenance branch, but it has created many layers of frameworks:

Buildbot->Bootstrap->TinderboxClient->MozillaBuildSystem

As you can imagine this can be quite nightmarish to debug and add features to. I believe strongly in backwards compatibility and incremental development, but the Bootstrap and TinderboxClient client layers are largely invisible to anyone outside of the Mozilla Build&Release team.

I think where we really want to be is:

Buildbot->MozillaBuildSystem

Wherever possible, Buildbot should do the same things a developer would do, and the configuration should be as clear as possible to read and modify.

I have some thoughts on how to get there on the wiki. The first step is to slot Bootstrap into place which is actually pretty easy as it just calls Tinderbox Client anyway. The larger work here is moving to the more direct "Buildbot->MozillaBuildSystem" scenario, for which I have a working prototype and it's configuration, if anyone is interested in seeing more.

Note that I'm not talking about changing Tinderbox Server or any of the existing mechanisms that developers use to clobber builds or check build status. bhearsum added Tinderbox Server and Bonsai support to Buildbot a while back, so builds show up on Tinderbox and we can configure them to be triggered only on checkin (as opposed to the continuous loop that Tinderbox Client currently does).

I have started a newsgroup thread in mozilla.dev.builds (subject: "Migrating Tinderbox to Buildbot"), please follow up there if you'd like to discuss.

EDIT - fix typo