For organizations wondering if they can utilize the agile methodology for large-scale development products, IBM's Scott Ambler offers this thought: Bureaucracy and discipline are two different animals.
Meaning, "A lot of traditional development is based on shaky theoretical foundations that don't reflect human behavior. It's not viable in many organizations," said Ambler, practice leader, agile development, with IBM Rational. "The agile community has a higher focus on quality and value; we're a little smarter about the way we work."
He added, "Many traditional organizations suffer from the misconception that bureaucracy equals discipline, but bureaucracy is bureaucracy and discipline is discipline."
In fact, asking if agile can scale is the wrong question, said Damon Poole, founder and CTO of AccuRev.
"The question to ask is, 'How does software development scale in general?' Traditional development doesn't scale very well anyway; you hear all the time of large projects that were cancelled or delayed. Scaling is independent of methodology," he said.
Does Poole believe agile scales better than traditional methodology? "Absolutely." However, he added, "There's a question of proven scalability. There are fewer proof points of agile scalability; that's just the way it is because there are fewer large projects. But look at the algorithm of agile development. There's more in it that allows you to scale."
In a SearchSoftwareQuality reader survey conducted earlier this year, more than half of the respondents (53.52%) cited agile team sizes of four to nine, while 22.54% cited agile teams smaller than three, and 19.72% of respondents cited agile teams of 10 to 20. Just 1.41% of respondents cited agile teams of 20 to 50, and 2.82% had teams over 50.
Perhaps reflecting the small amount of large-scale agile projects, just 16.9% of respondents cited scaling as a challenge to using agile, while more than half (52.11%) cited communication as a challenge, followed by documentation (47.89%) and resistance to change and tool integration (both at 23.94%).
Scaling agile using sub-teams
At IBM, Ambler said there are other agile projects on the order of 500 to 600 people. To manage large teams, both Ambler and Poole agree the project needs to be broken down into smaller components and sub-teams.
"There's no magic to this," Ambler said. "The architecture should be a system of subsystems."
He said the structure of the teams should be aligned with the architecture, and ideally the sub-team members should be co-located.
"The worst way to do this is to have the business analyst in Toronto, the test analyst in London, etc.," Ambler said. "You'd get a phenomenal amount of coordination required among these sites, across time and geographic boundaries, which increases cost and risk."
Poole is on the same page: "Making teams bigger is problematic; you don't want teams to get bigger than 10 or so people for any kind of development. Say you've got a product and you've got 250 people working on various aspects of it. You could say you have a team size of 250, or you've got 25 10-person teams and set it up so the teams are all working on something of appropriate size. To scale, you have to have a componentized software code base so people can work on different components and communicate API changes, and put team members in the same locations so you have good communication."
Making it scale
Both Ambler and Poole recommend practices to help better scale agile. Ambler said some upfront, lightweight modeling is key.
"Traditional development is based on the concept that you need to think through everything up front in detail, which is a wonderful theory, but practice shows that's not the case," Ambler said. "Detailed upfront modeling increases risk, and you end up implementing functionality [that] stakeholders don't want. With agile modeling, you get value from modeling, but you're not taking the hit of extraneous documentation and extraneous detail."
Ambler also said some extra coordination is required to scale agile. At the project level, "Scrum masters have to interact and make sure sub-teams are going in the same direction, and product owners of each team have to negotiate priorities to make sure teams are working in the right order."
There also has to be coordination at the architectural level as interfaces and requirements evolve, he said, to negotiate technical issues, "so architectural coordinators on each team have to interact."
One agile practice that Poole said doesn't scale well is continuous integration (CI), which is why he recommends multi-stage CI. With multiple teams all doing continuous integration, he explained, the impact on the teams and the mainline code grows exponentially. With multi-stage CI, "you as the individual developer don't get changes from everyone all the time and check in your changes all the time. You check in [code] when it's working and get other's changes when you're ready."
The idea is to compartmentalize, he said. "It's implemented with branches. With multi-stage CI you have a branch for each team, and they check in code like that's their mainline. If it goes well, you move the change on immediately, so you're moving as fast as you can but no faster. But if your team has a problem, it doesn't affect other teams."
What about those 3x5 cards and standup meetings? If a large team is split into smaller, co-located subteams, those methods can still work, Poole said, but they are clearly more problematic for distributed teams.
Agile practices in traditional environments
Interestingly, Poole said organizations that do traditional development can incorporate some agile practices that will help scale a project, such as breaking the project into smaller teams, co-locating teams, and holding stand-up meetings.
"I'm a fan of agile, but to me it doesn't matter whether agile is successful," he said. "It only matters if software development in general is more successful, and agile brings more tools to the table."