One of the things that was pounded into me while working at MoCo is the idea of having a bug tracker and using it. I literally can't work without one anymore. It's the first thing I really pushed for at my new job (they were using various ad-hoc systems for project management, but not a real bug tracker for the software dev side). I've realized that I just can't keep everything in my head, various notepads and text files, etc. and expect to get anything done, or let anyone know what my priorities are.

In return, I really tried to hammer in the idea of fast, automated release cycles. We spent a lot of time (and the release engineering team does still spend a lot of time) wrapping the build system and other tools so that they can be run and the output verified automatically, chasing that ideal of the Formula One-style hand-off to QA and to the users.

The way releases work now is incredible, just night and day from when I started at MoCo a little over two years ago. However, there's one thing that's always bugged me, and since I just had the opportunity to set up an automated build/release environment, I thought I'd expound a little bit on it.

The one thing is that nightly builds of Firefox just aren't the same as the release builds. The way updates work is different, branding is turned on, bits are signed (on Windows), the directory structure for files is different. Firefox releases are actually rebuilt from source for each release.

So what? None of these, even added up, are a big deal, right? Obviously releases work fine, and there are a ton of great people (and the tools they've made) that make sure that nothing is missed because of this. But wouldn't it be great if we could just take the nightly updates and builds that have already been put through the ringer by thousands of people, and give those straight to QA? Or if we can't have that, how about at least have the release builds put through the same tests and available to QA immediately after checkin?

Am I pushing some fanciful, architecture-astronaut utopian vision? I don't think so, because this is how I've done releases in the past, and this is how I do releases now. Let me tell you about it.

I use Hudson, which I can't recommend highly enough (well, if you're not allergic to Java, I guess). It makes this kind of process easy. It's not necessary to use it to achieve this of course, I'm just throwing this out as a data point.

On each checkin:

• a unique build number is generated
• a new build is generated (I also have it run unit tests, and install the software to run functional tests)
• release files and other artifacts like build logs are archived, and checksums of the files are stored
• if anything goes wrong, the team and the developer who checked in the latest change are notified

The software is available to QA as soon as this automated process is complete. When it's time to release, I can tag the build via the web UI (although it's easy enough to do outside of Hudson if you have the build number, which in turn contains the branch/datestamp/revision info needed).

Having the next release always "on tap" makes it easy for me to largely ignore the build/release side of things, and focus on developing software, writing tests, and tracking down problems.

Now, Mozilla's situation is way more complicated, which I alluded to a bit earlier. This post isn't a "see what I can do!" rant as much as a "look what's possible!" idea. I think that this kind of setup is totally doable for Mozilla's products, but there are some serious issues:

• branding is turned on at compile time. having nightly builds not called "Firefox" is a *good* thing, as otherwise end-users would be very confused.
• "--enable-tests", needed for unit tests, cannot be run in release builds at the moment (for technical reasons outside the scope of this post; I'm sure there are bugs on this)
• release builds are signed and have a different filename format and directory structure (e.g. "firefox-3.0.pre.en-US.win32.installer.exe" for nightly versus "3.0/win32/en-US/Firefox Setup 3.0.exe")
• release builds are cryptographically signed, to assure users that these files really were created by MoCo (regardless of what mirror or download site they may have come from).
• nightly updates are only for en-US, and use a different set of tools to generate updates, and a different mode of the update server to serve updates (some ideas for fixing this problems are in bug 410806, but again this is outside of the scope of this post)

So all of these are pretty much good things (branding, signing, etc.) or technical issues that could surely be fixed (nightly updates, unit tests). Arguably, nightly users and release users tend to be very different people, with very different needs and expectations, so all of the "intentional problems" here are really good things. This pretty much eliminates the possibility (as far as I can see) that Firefox release engineers could take a nightly build and be able to ship that as a release build.

Even if the branding issue were solved (e.g. repackaging), signing still needs to be done, partial diff files would need to be regenerated, and probably other things that I'm overlooking. The automated tests that were run on the nightlies may not be applicable (you may scoff at the paranoia, but there was a bug regarding the size of the Vista icon in official branding found late in the Fx3 beta cycle which caused a bunch of grief. This situation was improved by making a Minefield version of the same icon, which is a good fix, but I think my point still stands).

Here's another option - why not create a real, honest-to-god Firefox release build, on each checkin (or at least alongside each nightly build)? This at least makes it available to QA as soon as humanly possible, and it could probably be opened up somehow to interested community testers (human-triggered builds are right now, just put into a special area).

Maybe I'm just spoiled working on little tiny projects, but I think even the already super-fast and extensively tested Firefox releases could be made super-faster and the tests extensivlier, at the cost of freeing up the release engineers of the need to babysit the One and Final Release Build.