I will be on vacation from October 16th through the 23rd in Toronto, and giving a lecture at Seneca on the 20th. If you are in the area and want to hang out, feel free to give me a buzz.
The release automation work is at a reasonably happy place, I have managed to write and test the following steps, and post patches for review (the high-level steps are described in the app's README file). :
* Tag
* Build
* Source
* Repack
* Updates
The Stage step is actually mostly tested, but I keep running out of disk space on the staging machine, so I'll need to get creative on that one. Sign is fairly simple and mostly manual (which is desirable), and Release fairly simple (but obviously critical!) - it's the act of copying the staged/signed bits to the official release directories.
One thing I feel I must mention is that this tool does not necessarily support what we consider the ideal process - it instead supports the process that we use, and that is known to work.
However, it is difficult to introduce benefitial changes and explore alternatives since we haven't had a good staging environment or set of verification tests to make sure that we haven't introduced any undesired side-effects.
The trickiest bit of this isn't so much the steps themselves as having some kind of automated verification that the step succeeded so we can trust that running the next won't be a waste of time.
Our current process is very human-time-intensive, since a release engineer needs to kick off and verify each step, and some of the steps take several hours by themselves (builds and update generation/verification, primarily). If something goes wrong (due to an unexpected change in the product, a bug in one of the tools, or just Murphy's Law) then we need to determine the last "good" step and restart from there.
Automated verification does of course have a point of diminishing returns, and Mozilla-based products are complicated enough that this doesn't really provide any direct QA benefit, besides not wasting our tester's valuable time on something a dumb computer can catch (like a bad tag, bad build, mismatched or nonexistant update paths, etc.).
The other big downside to a human operator being the default is that humans function much better with sleep and time off (prolonged focus being bad for overall concentration) and it's a bad use of creative energies. An automated process doesn't need to pause between steps, and won't introduce variation through attempts at creativity. The place to be creative isn't in the scope of a release, but in thinking about and improving the overall process (generally best done in between releases, based on the lessons learned from the past).
It should of course be possible for a human to jump in and drive the process if needed, especially fixing and rerunning steps which failed for an intermittent reason, bug in the tools, etc. It should hopefully not be the norm, but it's a reasonable use case for this kind of tool.
The ideal use case that I can think of right now would be: "code is frozen; declare and obtain sign-off for names/numbers/locales/etc. and kick off the release process". Respinning to pick up source changes is acheived by a variant of the Tag step, and the process is restarted and runs through the same Build->Stage steps until we're happy with it.