Over the years, we’ve taken pride in making our tools, product and development process, as open and transparent as possible. Our tools (Bugzilla, Litmus, Wiki) are publicly accessible, our source code open (svn) and we’ve blogged on many occasions about our Agile practices.

Today we’re taking a step further by publishing our Development Survival Guide. This presentation is an internal step-by-step guide that we take new engineers thru during their orientation. It’s a good summary of what to expect on a day-to-day basis as an engineer working at Songbird (other than daily Fussball tournament and unfettered access to the beer stocked mini-fridge). It’s your opportunity to take a peek from within.

Want get even closer? Apply for one of our openings.

Download Songbird Development Survival Guide 1.2 in pdf.

Fortunately, the metrics gathered during each Agile release cycle can be very helpful for that purpose. Once we gain a good understanding on what is being worked on, for how long and by how many people, we should be able to extrapolate this to forecast future releases.

Let’s take a look at what activities take place during a typical release cycle:

1) Plan release
2) Write code
3) Test
4) Fix bugs

Then repeat ad nauseam.

In that context, the programming tasks can be categorized as follow:

1. Planned work

This is the body of work identified during the planning stage. This is the raison d’être of the release. For the most part, this covers new features or less tangible things such as performance improvements. This is what we’ll want to talk about when the product gets released. It’s an easy planning trap to think that this is the only work required.

2. Unplanned work

As the name implies, this is work that was unforeseen at the beginning of the release but is required to be completed before the product can ship. It can be further refined as follow:

a) Change in requirements
This should not be unexpected. In fact, any Agile methodology assumes that there will be changes down the road. This is not a problem per se as long as there is a mechanism to trade features, extend duration or increase resources.

b) Omissions in planning
Because the planning period is relatively short, it is assumed that not everything is fully specified or researched upfront. As development proceeds, new pieces are discovered and introduced into the plan.

3. Bugs

Defects can be considered a side effect of software development. They can affect the product in different ways:

a) Regression
These are defects introduced when working on new code. They usually impact unrelated functionality that used to work before.

b) Defect in new feature
These are problems in a newly coded feature. The feature does not work quite as expected.

c) Existing bug
These are bugs present in the previous version of the software. They are either known or newly discovered during the course of the release and prioritized to be fixed now because they affect existing users.

Learning from the past to project in the future

A believable long term plan needs to layout new features on a timeframe against a set of resources. Because there is only a limited amount of time and people allocated to thinking through the issues upfront, the initial estimates are always very rough. It is then very difficult to predict how much time certain things will take or how many people will be required. To increase accuracy, you’d have to invest more time and resources, which is not practical, mostly because these are the very same resources that are critical to deliver against the current release plan.

You need a model to help forecast based on imperfect data. You could decide to simply pad all the current estimates, but you’d still need to figure out a factor that’s sufficient.

A better model is to find a way to anticipate the additional work that will be generated by the introduction of a brand new feature. With that model, we can layout a plan that should allow sufficient room in every release to accommodate for all the work, planned, unplanned and bugs.

Let’s take a look at historical data from previous Songbird releases:

The first row in the table contains the budgeted points for each release. That number represents the sum of all planned work as determined at the end of the planning phase for that release. The release schedule was set based on that number and the historical team velocity.

The second row represents the total actual points completed during the release, including features and bugs. This represents that amount of effort to get the release out.

The third row represents the actual completed feature points only. Next row computes ratio of planned vs. actual work.

It’s very tempting to compute a velocity based on total completed points and use it as a predictor for how much work the team can accomplish. While it’s true that it is a measure of total effort, it can’t be used against planned work only.

What we want to determine is that given 1 point of planned feature work, how much unplanned work and bugs will be generated and thus how long will it take to complete. Then normalize this for team size and you get a factor that can be used for long term planning.

To get there, I’ve computed a velocity for each release based on planned work divided by actual release duration (in days) and number of engineers. I then use the median value across release as a factor to turn estimated planned points into calendar days. With that in hand, I created a simple model where scope, duration and resources are linked. Set two values and the third one gets computed automatically. This model becomes very handy to run through what-if scenario during roadmap sessions. It provides answers to business questions such as: “How long would it take to do feature X?”, “Can you meet deadline Y?”, “How many more engineers would it take to do Z in Y time frame?”.

Hope you enjoyed learning more about our approach to the difficult task of long term planning. I’d be interested to hear about how you’ve approached the problem, so feel free to share your story.

Lastly, let’s remember the words of the famous Danish physicist Niels Bohr:

“Prediction is very difficult, especially if it’s about the future.”

Songbird Release Trains

We concluded that a dashboard that was more dynamic and worked in real time with Bugzilla would provide a more accurate picture of development progress. This is an overview of the couple of extensions we added to the tool.

Dashboard

The dashboard focuses on the current iteration (all P1 stories/tasks/bugs for a release target milestone). It provides a transparent and dynamic view of what’s under active development for the week.

Items are grouped by developer so we can easily review what each person is working on in this iteration. There are few additions that make it more useful than a plain Bugzilla query. When an item is carried over from a previous iteration, it gets highlighted, providing visibility on something that may require more work than anticipated or might be blocked by some dependencies. That kind of view was lacking before and it made it difficult to figure out what was happening when tasks got carried over. Similarly, when a new task gets added to the plan mid-iteration, it’s identified as such, making the tracking of intake very easy.

Because of our code review system, often time tasks are completed from a programmer standpoint but can’t be checked in until a positive review (R+) is performed by someone else. Additionally, some code requires the re-compilation of large amounts of 3rd party libraries before the patch is available. This kind of information is now bubbled up on the tracking page so the exact status of a particular issue is visible at a quick glance.

Iteration Dashboard

This dashboard is helping us to know what everyone is doing at any given time. It fosters a sense of accountability and ownership of the issues and provides a sense of accomplishment when the work is completed.

Below is a view of the legend for the Dashboard. Most attributes for an item are extracted from Bugzilla and represented via an icon.

Dashboard Legend

Punchlist

Once a release reaches QA, a release branch is cut and the focus of the team switches from working on planned features to fixing bugs. We found that this period is best managed by using a punch list which contains all the issues remaining to be resolved before the release can ship as opposed to a weekly plan of what gets accomplished for the period. It keeps the team focused on the entirety of what’s on our collective plate. During that period, issues are being promoted to Blocker status to represent what is blocking the next milestone (e.g. Beta, RC, Final). It also signals what can be landed on the branch automatically and what needs further clearance. This allows us to throttle the rate of change being introduced into the source tree so that QA can be ensured stable builds.

Release Punchlist

When in punchlist mode, it’s critical that all issues are being assigned. The list reflects that by highlighting items that have no owner yet.

Pretty Graphs

Two frequently tracked Agile metrics are Burndown and Velocity. Those are now being graphed automatically for each release. The Burndown helps forecast when the work will be completed. The Velocity gives us a historical perspective on what the team was able to accomplish and thus helps us plan better.

The Burndown graph breaks down tasks (what we refer to as “planned work”) from bugs (unplanned consequences of software development). When the Burndown line cross the horizontal axis, we’re done. Notice in the graph below the bug points going up during the QA period.

Burndown Graph

The Velocity graph plots the team velocity, normalized per day. Actual Velocity is computed at the end of each iteration and plotted along side the Planned Velocity for easy comparison.

Velocity Graph

These new tools provide a dynamic view of the state of the release. They bring transparency into the development process for everyone involved.

This is a repost of a series of article I originally published on Songbird’s blog

In the previous two installments of this series on Agile development at Songbird, I’ve covered our move from waterfall to Agile and provided an in-depth look at some actual release cycles. In this last post, I’m going to introduce a tool – which I gave the uninspiring name sdpbot – built internally to help facilitate the tracking of our releases.

Wrestling Bugzilla into shape

Because so much of our existing workflow occurred in Bugzilla, we’ve decided to use it as a central database to drive our process. Every actionable project artifact lives in Bugzilla, a Feature, Story, Task or Bug. From a release standpoint, the only actionable items are Story, Tasks or Bug and hence, we only track these. Here is how we’ve organized our bugzilla.songbirdnest.com to help us track each release.

Release train

Every release train has a name and is used to create a target milestone in Bugzilla. This allows us to put items in release buckets. See some examples of what was included in the Fugazi and Genesis releases.

Priority field

As we create stories, tasks and bugs we assign them to the release milestone and place them in a priority bucket. Each priority level indicates how the item is being treated during the release:

• P2 – must have for the release, next candidates for promotion to P1
• P3 – nice to have for the release, we may ship without it
• P4 - nice to have and low risk. This represent the kind of work that we probably will undertake during the QA phase as it is low risk and won’t likely cause a regression.
• P1 has a special meaning. It is used as a flag to represent what is being committed by the team for the current weekly iteration. Everything in P1 signals our tool that this is our “plan” for the week and is under active development (We are considering changing this in the future to use the assigned bug functionality since Bugzilla 3.2 will now allow you to accept and assign a bug in one operation).

Costing

Weekly iterations

Every Monday, we plan our work for the week. Every member of the team elects items to P1 status. We then tally up the total points and determine if the plan is sound. Once we are all in agreement, the plan gets locked. This simply means that the tool takes a snapshot of all the items marked as P1 and saves their ids.

Daily reports

The main function of the bot is to compute delta and generate a daily report. Let’s take a look at every sections of the report.

Iteration

A simple number to represent the iteration (week)

Plan

This section represent the plan of records as snapshot at the beginning of each iteration. Once the plan is locked, it becomes immutable.

Duration

Duration of the iteration. Usually 5 working days, but it automatically adjusts for holidays.

Remaining

Total points for P1, P2 and P3 items which status is open (e.g. UNCONFIRMED, NEW, ASSIGNED or REOPENED)

Planned

Total points for P1 items that are opened.

Planned Velocity

Normalized points per work day.

Actual

This section represent the actual metrics computed at the end of each iteration. The iteration ends on Sunday at midnight.

Intake

Represent everything that was either added or removed to the P1 list (the plan) during the iteration.

Intake Velocity

Normalized intake points per work day.

Completed

Total point of everything that was marked fixed during that iteration.

Team Velocity

Normalized completed points per work day.

Carried over

Everything planned at the beginning or added during the week and not completed by the end of the iteration.

Environment

Originally, this was introduce to track distractions outside of the core release that could impact the team velocity. The idea being that it would give us a gauge of what the “environment” was like during that period. This columns would capture everything open or closed in the special ASAP Bugzilla milestone. We used this classification to track urgent issues, infrastructure work, etc. As our process matured, the need for this diminished and we no longer use this.

Here is a sample of such report for Fugazi

What to look for

Every day an email report is being generated allowing us to track progress to date. For every column described above, a link is provided. The link points back to Bugzilla and contains a list of ids for the items included in the list. Every item can then be reviwed independently.

At our weekly planning meeting, we usually start by reviewing what was carried over from last week. Ideally, we want this to be a null set, but the reality is that it often contains a fair amount of items. It’s primarily due to intake during the iteration. Items are being added because of regression introduced and discovered, new tasks being identified or slight adjustment to the plan. This is definitely an area we need to improve on.

The next view is the planned items for the new iteration. This allows us to make some tweak to our plan before we commit.

The intake views are very useful to analyze what is being added or removed during the cycle.

The team velocity gives us a measure on how well we are doing and how much load we think we can carry during the next iteration.

Finally, the remaining points tally gives us our burn down number. This is how much work is left and by dividing this with our historical velocity gives us a sense of how long it will be until we are done.

Open source library

SDPBOT is built as a Ruby library. The source code has been open sourced under the MIT license. You can download it here: http://code.google.com/p/sdpbot/

We have made no attempt to make it generic, so do expect to do some work to customize it for your own needs. If you’re using Bugzilla and want to generate some type of reports you will hopefully find the library to scrape and instantiate objects from Bugzilla very valuable. Enjoy!

This is a repost of a series of article I originally published on Songbird’s blog

Previously, we’ve examined the new development practices that the Songbird team adopted to plan and track a release. Everyone on the team was very eager to put them to the test. Unfortunately, at the time, we were still in the middle of the 0.3 release cycle and new work could only be started once that release was completed. During the 0.3 release, everything was still treated as a bug, but in fact, many bugs were stories and tasks in disguise. We decided to apply some of the newly defined tracking principle to help us guide and finish the cycle, so we could start fresh with our next release as soon as possible.

Cuánto es?

The first step was to add cost to everything. We introduced a new cost field in Bugzilla and put a cost value on everything according to our new scale of 1, 2 and 3 points. With costing in place, we were in a position to compute how much points the team was able to complete in a typical work week. That total, normalized per work day became our team velocity.

Below is a chart representing our velocity over many — one week — iterations during the 0.3 release (code name Bowie). The blue line is the number of points the engineering team completed, averaged per work day. The red line tracks the cost of new things being introduced, normalized per work day. The green line tracks the net velocity.

It quickly became apparent that as the team took things off the pile, new work was being identified and added. We had to keep track and take this into account. We named this intake to globally represent new functionality, regressions and newly discovered bugs.

The net velocity gave us an indication on how well we were doing overall. When it gets in negative territory, we are losing ground.

You can see below 3 events that had clear impact on intake, namely scope creep (some features were not well defined upfront), and bug intake due to public feedback from a blessed build and a release candidate.

Also noticeable is a week when the team was not as productive as usual. With that information in hand, we were able to have open conversations about the state of progress and try to determine the cause for it. Sometimes, a low velocity is simply because work gets accumulated in a week and does not get checked in until the next. We call this carry over. Other time, there are some inevitable distractions such as a move, interviews, equipment failure, etc. In other cases, the team is just having a bad week.

With this in place, we were able to understand the rate at which the team was completing work. We created a burn down chart that tracked the total points of known work left. Using the team velocity, we could forecast an expected release date with a simple best fit projection. When the line crosses the X-axis, you are done and can ship.

If you compare the two charts, you can see how much influence intake had on the release. This became a key component to take into account in our planning. By budgeting points towards intake, it allowed us to reserve some engineering capacity to take change into account upfront and have a more realistic schedule.

Where does intake come from?

By formally tracking our intake, we were able to better characterize the nature of change. Most of our intake comes from change introduced when features start to materialize in the product and can be tested. This is a desirable effect of adopting an Agile practice. Other contributors are defects being reported by existing users, which is a benefit of early releases. Less desirable intake come from regressions or new tasks that resulted from bad assumptions or misunderstandings. Those can usually be mitigated by increasing unit tests coverage and more upfront detailed planning.

Full Agile cycle

Let’s take a look at another release cycle. Dokken was our second release in which we used the new process from inception. The charts below represents velocity and burn down respectively. Note that the scale on the velocity has increased. There is more dynamic range. Because the release started from day 0, we noticed a ramp up in the velocity. We learnt not to be necessarly alarmed by this. In a new release cycle, the team needs some time to “prime the pump” of development. As the release progressed and we got better visibility of the team progress, we decided that we should defer feature work that was identified as nice-to-have. This is another thing we did during planning. We prioritized work in must have for the release, hope to have for the release and nice to have – mainly cosmetic changes, low risk bugs – buckets. This gave us a pre-negotiated way to easily shift features as the release progressed.

By actively tracking and managing the intake, we were able to steer the release and deliver within weeks of the projected date. Not great, but better than our previous releases. Dokken was a particular difficult release as we undertook a lot of device support work, which can lead to nasty device compatibility problems.

Embracing change

The next release, named Eno presented some interresting characteristics. The intake was relatively high thru the release but the team was also maintaining a higher velocity. This is a good example of the team achieving a good level of agility. Change is being introduced throughout the release and the team is well prepared to tackle it. Also notice that the spike in intake due to feedback from release candidate has a corresponding level of increase in team velocity. This is due to the shortening of feedback loop. With proper ownership, the developers are able to respond in real time to issues being discovered.

Notice how the burn down trend is converging almost linearly. This means that the team is achieving a sustainable pace, leading to more predictable ship date.

Achieving high velocity

Our latest release, Fugazi is another example of a successful cycle. This was the shortest release cycle the team ever undertook, with only 4 weeks of planned development work and 3 weeks of QA. In order to maintain our original release date, we had to defer some lower priority work in iteration 3. Despite the shorter release period, the team actually completed more points per iteration than any other release. We maintained an unprecedented high velocity, which in turn allowed for a high level on intake to be absorbed, so all the planned must have featured could be kept in the release and still hit our original target date.

DIY tools

In the last part of the series, we’ll cover the tools we created to make the tracking easier and how they are being used on a daily basis to help us steer the release. Stay tuned.

This is a repost of a series of article I originally published on Songbird’s blog.

This is the first post of a 3 part series presenting our experience moving Songbird development to an Agile process.

Drowning in the waterfall

Up until version 0.3, Songbird development had been following a fairly traditional waterfall model. Realizing the ambitious vision of building both a platform and a desktop media player has presented many challenges. A lot of plumbing infrastructure is needed before any features can be created. Faced with that challenge, the engineering team did what engineers do best, they designed a very comprehensive system, planned for it very carefully and started cranking code.

During the planning phase, the team estimated the work to the best of their abilities and a Gantt chart was created to reflect identified dependencies and track progress. Unfortunately, this approach led to lengthy release cycles (10-12 months) with lots of room for scope creep. When the release finally got completed, lots of good work was accomplished (over 1200 issues where addressed in 0.3 alone) but the lack of visibility was problematic and the slow pace of releases was too demoralizing.

We recognized that the schedule was build on assumption that we knew everything upfront. There was a sentiment that the whole Gantt thing was a little removed from the actual work and that overall “things were going ok, because we were kind of tracking it” was not an acceptable way to run our project. We had to accept that our planning and scheduling practices were broken.

Besides the effect on morale, a long development cycle presented the following problems:

• Nothing can be released until everything is completed and put back together.
• Difficulty to bring visibility on real progress (i.e. when can we ship?).
• Last stable release branch and trunk drift apart significantly. Especially problematic to support commercial partners.

We also felt that our task estimates were not granular enough and the built-in slack was not taking into account other work. Unfrequent build also meant a lack of visibility on product quality.

That kind of development approach also fosters poor engineering and product development habits, such as:

• Reduced sense of urgency to release code. Discipline drops, unit test failures increase, bugs pile up. “We have enough time to fix this” mentality develops.
• Ultimate release becomes a huge effort as the routine around releasing code does not form.
• No continuity in QA, bugs accumulate in a backlog.
• No context for new work, so there is a tendency to add more things to existing release, also known as feature creep.

Becoming more Agile

We decided to adopt a new approach to development that would address those issues. These are the objectives we tried to fulfill:

1. Satisfy our customers (end-users, developers, partners) through early and continuous delivery of valuable and innovative software.
2. Provide ability to react quickly to business changes.
3. Reduce product defects and security exposures.
4. Ensure team and product sustainability. Focus on ease of product maintenance, allow the team to maintain a constant pace indefinitely and be resilient to turn over.
5. Provide good visibility into progress and release date.

We felt that we had a good foundation of existing practices we could build upon. Build automation, continuous integration, unit testing, peer review process for code commits, automated api documentation and the use of Bugzilla were already well established within the team. We needed to augment those with a few others borrowed from Agile methodologies.

We added the following set of guiding principles:

• Reduce release length and maintain a releasable product at all time.
• Working software is delivered frequently (weeks rather than months) and is the principal measure of progress.
• Provide rapid feedback loop to QA and Product Manager.
• Develop a system for better estimation and tracking.

Reduce release length

At a high level, we’ve committed to release a major update of Songbird every other month. We opted for calendar driven releases with a short development cycle of 4 to 6 weeks of coding maximum. We adopted a naming scheme for each release train drawn from a music artists theme. Alphabetically sequencing those release trains makes it easy to see the order of releases (Eno release comes after Dokken for instance).

Measure progress against working code

Within a release cycle, we introduced the notion of iterations of 1 week length that we used for planning and measuring progress against. This gave us the context to improve our estimation and tracking system.

Develop a system for better estimation and tracking

Estimating how long software development takes is one of the most difficult thing to do.

We created a High Level Roadmap that captures the scope and sequencing of each release.

We came up with a set of lightweight artifacts that would help us represent and track what needs to be done.

• Feature – High level product feature (bullet point on product box, if we had one)
• Story – Smallest increment of implementable end-user value
• Task – Engineering implementation detail, chore, etc.
• Bug – Defect in existing functionality

In support of that, we also created on our wiki design docs that represent wireframe and accompanying notes to explain each feature. You can see an example of such document for the view menus clean up in Eno release.

For each story, task and bug, we’ve introduced a costing system based on points. The scale starts as at 1 point, for something easy that an engineer can do in a day. Two points means it will take some time to think about it and write the code. 3 pointers are reserved for the most difficult issues, requiring research or lots of code. The idea is to come up with a normalized unit of work across the team that we can measure ourselves against. Moving from a time based estimate to point base is not easy. For one, the scale is not linear (e.g. 3 pointers don’t take 3 times as long as 1 pointers). However, the basis for this is to simplify and reduce cost estimate process to a good enough level that we can measure and forecast. While it may be a blunt instrument, there is diminish return in spending more time refining and tracking estimate at a much more precise level.

The key benefit from developing a consistent costing practice is the ability to measure progress and use it to forecast future work. Entering Velocity. By computing how many points get completed by the team over an iteration, we were able to normalize the output in the form of a velocity metric. This gives us a sense of how much stuff we can accomplish in a typical cycle.

To track our progress, we setup 2 iteration meeting, one to review and plan the iteration and the other to steer the iteration at mid-point.

All the artifacts and measurements are tied together on a Release Plan published on our internal wiki.

Feedback loops

Another important factor is the acceptance and validation of work. In order for points to be counted in an iteration, there needs to be an agreement on when things are “done”. Nightly builds are being used for verification every day. On the bug front, QA validates engineering fixes and mark them validated. Similarly, Product Manager accept stories as completed when they are satisfied with the implementation of a feature.

Putting it to practice

With all these new practices in place, we were eager to tackle our first Agile release known as “Cher”. Our next post will take a look at the roller coaster of real life release cycle.