Yesterday one of our engineers in Europe sent me a link to a white paper on Dr. Dobb's. The white paper, Top 5 Software Development Process Challenges produced by AccuRev, makes some very good points about the complexity of managing software changes via Software Configuration Management (SCM).
These two paragraphs really resonated with me:
In any particular release, it’s common to take all of the "done" code and merge it to a release branch. There is a set of "un-done" code mixed in with this code base.
In addition, with shorter release cycles, merges that needed to be done once every few months now have to be done every two weeks, or every month. The pain of merging is now persistent throughout the development process.
This is exactly the problem our development team faced two years ago. As our team grew and we added developers in different parts of the world, we faced the challenge of how to get to a stable release point where the features that we wanted to release were integrated and tested by the release target date. In addition to this challenge, merging the changes from different developers became a huge overhead for us. We tried having everyone merge their own changes and we tried having a Merge Meister, but neither worked efficiently. Finally, we changed our work flow to use feature dependent SCM branches. We went from three baselines to more than 100 over the last two years. All of the "feature branches" are built and fully release tested nightly. Multiple stake-holders collaborate on each feature branch including: the product manager, the feature architect, multiple developers, and QA. All stake-holders make their commits to the same "feature branch". This results in well understood and atomic change sets that eventually get merged into the "release branch". Because of this workflow change, we can now literally do a production release on any day that we choose. Feature branch change sets only get merged into the release branch after they have independently passed testing of the new functionality as well as full functional release testing. This workflow change required a significant investment in hardware infrastructure, but the improvements in the productivity of our team are dramatic.