You are on page 1of 11

November 2004

An Introduction to Agile Software Development
by Victor Szalvay, co-founder
Danube Technologies, Inc.
Web site: http://www.danube.com
Web log: http://danube.com/blog/
12011 Bel-Red Rd. Suite 201
Bellevue, WA 98005
(425) 688-0888, ext. 812

Introduction
This paper is an introduction to the Agile school of software development, and is primarily
targeted at IT managers and CXOs with an interest in improving development productivity.
What is Agile? How can Agile help improve my organization? First, I introduce the two broad
schools of thought when it comes to software development: traditional sequential, a.k.a. “the
waterfall method”, and iterative methods of which Agile is a subset. My objective is to
demonstrate the short-comings of the waterfall approach while providing a solution in iterative,
and more specifically, Agile methods.

Part I – Shortcomings of Traditional Waterfall Approach
The essence of waterfall software development is that complex software systems can be built in a
sequential, phase-wise manner where all of the requirements are gathered at the beginning, all of
the design is completed next, and finally the master design is implemented into production
quality software. This approach holds that complex systems can be built in a single pass, without
going back and revisiting requirements or design ideas in light of changing business or
technology conditions. It was first introduced in an article written by Winston Royce in 1970,
primarily intended for use in government projects1.
Waterfall equates software development to a production line conveyor belt.
“Requirements analysts” compile the system specifications until they pass the finished
requirements specification document to “software designers” who plan the software system and
create diagrams documenting how the code should be written. The design diagrams are then
passed to the “developers” who implement the code from the design (See Figure 1).
Under the waterfall approach, traditional IT managers have made valiant efforts to craft
and adhere to large-scale development plans. These plans are typically laid out in advance of
development projects using Gantt or PERT charts to map detailed tasks and dependencies for
each member of the development group months or years down the line. However, studies of past
software projects show that only 9% to 16% are considered on-time and on-budget2. In this
article, I attempt to summarize current thinking among computer scientists on why waterfall fails
in so many cases. I also explore a leading alternative to waterfall: “Agile” methods that focus on

Copyright © 2004 Danube Technologies, Inc. All rights reserved.

Inc.999% of the time.000 purchases per hour. the requirements are the features and specifications of the system. design. Test 5. scalability and concurrent user support. consider the areas that must be addressed: business rules and exceptions. user roles and restrictions. In fact. All rights reserved. Deploy Analysis Design Phase Up-front Requirements Analysis What are requirements? From the stakeholder’s perspective. or the system must be accessible 99. What if your business needs are still emerging and certain aspects of the system are rapidly changing or cannot be defined yet? Business climates and objectives often change Copyright © 2004 Danube Technologies. . Architecture & 3. By virtue of a requirements change. and in many waterfall projects this can be very costly. Page 2 incremental and iterative development where requirements. browser or OS support. Requirements 2. Once finished. implementation. Code 4. which will in turn affect any implementation and test strategies. it is inevitable that attempts at up-front requirements specification will leave out some very important details simply because the stakeholders cannot tell developers everything about the system at the beginning of the project. The cost of change in a waterfall project increases exponentially over time because the developer is forced to make any and all project decisions at the beginning of the project. Requirements define what developers are to build. the system must have a web site with e-commerce capability that can handle 10. Do you think you could provide every last detail the developers need to know right off the bat? I have yet to encounter such a customer and I am hard pressed to think I ever will. In Figure 1. As a start. the intricately planned design can be affected dramatically.3 This means that the requirements typically change outside of the requirements phase in the form of “change orders”. the SRS is sent over the fence to the designers while the requirements analysts go to work on the next project. Analysts slave for weeks or months compiling everything they can gleam about the proposed system into comprehensive “Software Requirements Specification” (SRS) documents. One of the biggest problems with waterfall is that it assumes that all project requirements can be accurately gathered at the beginning of the project. and testing continue throughout the project lifecycle. user interface standards. the first block represents the requirements analysis phase of a software development project. Figure 1 Traditional Methods: sequential phased approach Project 1. For example. Imagine a scenario where you engage a software group to build a critical software system.

But is it possible Copyright © 2004 Danube Technologies. Because the foundations of software development are inherently unstable and unreliable.. The waterfall methodology assumes that up-front planning is enough to take into account all variables that could impact the development process. programming languages. functional software. organizations developing software must realize variables exist that are largely outside of management control. I often use a “drawing” analogy to help explain this effect. however. . This is entirely unnatural because customers find it difficult to specify software perfectly without seeing it evolve and progress. has no laws or clear certainties on which to build. It is therefore fair to say that software development is more akin to new product research and development than it is to assembly-line style manufacturing. the types of questions that would appear on the test were unknown when we started development. Markets are forcing the software development community to respond with flexible development plans that flatten the cost of change.g. All rights reserved. Bridge building relies on physical and mathematical laws. discovery. Also consider that the building blocks of software projects is usually other software systems (e. and artistry. changing. Inc. cookie- cutter solutions4. But this is what waterfall asks customers to do: specify the entire system without having a chance to periodically see the progress and make adjustments to the requirements as needed. and contingency. software is almost always flawed or sub-optimized. Although I’m a terrible artist. Since the test itself had not been released yet. The “I’ll Know it When I See It” (IKIWISI) law says that software development customers can better describe what they really want after seeing and trying working. Software development. As a result.). etc. when I draw a picture I need to see the drawing as I progress. waterfall projects allocate copious effort detailing every possible risk. The problem of undefined. the requirements are solicited from the user and some time later the finished product is presented to the user. But the system had to be done shortly after the tests were released. a national test preparation organization commissioned my company to build a simulator for an upcoming standardized test. mitigation plan. People need to see and feel something before they really know what they want. All software systems are imperfect because they cannot be built with mathematical or physical certainty. Waterfall is an “over the fence” approach. each foray into a development project presents new and difficult challenges that cannot be overcome with one-size-fits-all. database platforms. Page 3 rapidly. and emerging requirements presents a very large challenge to waterfall projects because by definition all requirements must be captured up-front at the risk of costly changes later. it would prove far less successful. and those systems that act as building blocks contain bugs and cannot be relied on with certainty. Software development is innovation. Can you afford to lock your business into a rigid long-term project where the cost of change grows exponentially? For example. especially in today’s age of instant information. In fact. Software Development is more like New Product Development than Manufacturing Software development is a highly complex field with countless variables impacting the system. If I tried to close my eyes and draw the same picture.

Winston Royce (of waterfall process fame) later noted that his ideas were incorrectly interpreted and that a “single pass” framework would never work (his article actually advocates at least a second pass). medical researchers could simply plug variables into equations to discover new medicines. On the contrary. Almost no software system is so simple that the development can be entirely scripted from beginning to end. in the 1990s sequential. The idea of revisiting phases over and over is called “incremental and iterative development” (IID). when used sequentially. The inherent uncertainty and complexity in all software projects requires an adaptive development plan to cope with uncertainty and a high number of unknown variables. . with IID requirements is an ongoing process that is periodically revisited. Each Copyright © 2004 Danube Technologies. Inc. sequential approach and implementing an iterative approach. where prototypes were developed for short-term milestones (see Figure 2). 6 The results were a dramatic improvement in cost and development time to market and ultimately lead to the popular rise of “lean development” and “just-in-time manufacturing”. This concept of iterative development hails from the “lean development” era of the 1980s described above where Japanese auto makers made tremendous efficiency and innovation increases simply by removing the phased. Honda. For example. supplanted the sequential “Phased Program Planning” (PPP) approach to new product development with a flexible. As new requirements surface and as the scope changes. the second is Y. HP. and the result is always Z. It’s ironic that a cutting edge technology field like software is so far behind more traditional engineering fields in terms of development methods.8 IID allows for multiple “passes”. Part II . result in a successful project each time.7 But longstanding insistence from IT managers to categorize software development as a straightforward assembly line progression has kept the software industry from evolving to better methods. All rights reserved. Fujitsu. waterfall-style approaches to new product development were effectively abandoned outside the software development industry. holistic approach where the traditional phases of development overlap throughout the product lifecycle. IID processes continually capture the requirements iteration after iteration Interestingly. and NEC. or iterations. Canon. Page 4 to predict any and all variables that could possibly affect a software project? The empirical answer is “no” considering the limited success of waterfall projects.Iterative and Agile methods Incremental and Iterative Development The simple ability to revisit the “phases” of development dramatically improves project efficiency. The first step is X.5 Waterfall therefore equates software development to an assembly line. since the late 1970s product development companies lead by Toyota. Can research really be relegated to a series of steps that when performed in sequence result in a new product? If this formulaic approach were adequate. the benefits of which other new product development industries have been reaping for decades. 3M. The development lifecycle is cut up into increments or “iterations” and each iteration touches on each of the traditional “phases” of development. defined processes can be established that. Following the lead of Japanese auto makers. over a project lifecycle to properly address complexities and risk factors.

It is the role of the automated test suite built around the rapidly evolving code to act as a harness that Copyright © 2004 Danube Technologies. like rapid production and minimum up-front design 15. "The New New Product Development Game". Jan. Takeuchi and I. Nonaka. Rather than focusing a lot of effort on big up front design analysis. 1986. Agile visionary Kent Beck challenged the traditional cost of change curve evidenced by Barry Boehm11 over twenty years ago. Agile methods: Embracing Change Agile methods stress productivity and values over heavy-weight process overhead and artifacts. and they further increase the iterative nature of the software lifecycle by tightening design-code-test loop to at least once a day (if not much more frequently) as opposed to once per iteration. Beck’s model espouses that the cost of change can be inexpensive even late in the project lifecycle while maintaining or increasing system quality12. Inc. Figure 2 Iterative approach: Overlapping phases of development Phase 1 2 3 4 5 6 Source: Adapted from H. the requirements.. This “concurrent” development approach created an atmosphere of trial-and-error experimentation and learning that ultimately broke down the status quo and led to efficient innovation.9 Although direct analogies between industries are never seamless. All rights reserved. 137-146. Agile methods promote a number of engineering practices that enable cost effective change. Nevertheless. Spiral. the success of lean development has influenced a broad class of “iterative” software methods including the Unified Process. . Harvard Business Rev. small increments of functional code are produced according to immediate business need. Beck’s idealistic “flat” cost of change curve has since been revised and softened by Alister Cockburn 13 and Scott Ambler 14 to reflect modern corporate realities. and implementation cycle was revisited for each short-term milestone. “Test driven development” is a quality-first approach where developer tests (called unit tests) are written prior to the functional code itself. Author and speaker Martin Fowler describes testing and continuous integration as the “enabling” Agile practices that allow for the advantages gained. was written and signed in 2001 although Agile methods have existed since the early 90s. Agile ideals can be applied to reduce the cost of change throughout the software lifecycle even if the cost of change is not perfectly flat. Agile methods promote an iterative mechanism for producing software. Page 5 phase was actually a layer that continued throughout the entire development lifecycle. and Agile methods. a concise summary of Agile values. pp. The Agile Manifesto10. Evo. design. To accomplish this “flatter” cost of change curve.

Scrum holds that straightforward defined processes alone cannot be used to effectively manage complex and dynamic software projects.16 As a result. the manufacturing industry has long known that certain chemical processes. in the Scrum process. are too difficult to script and define. Risk factors and emerging requirements complicate software development to a point where defined processes fall short. Page 6 allows developers to make aggressive code changes without fear of undetected regression failure. In fact. for example. even late in the project lifecycle. changing software projects. Agile project management approaches balance the four variables in software development while keeping in mind the limits associated with new product development. introduced the concept of empirical process control for the management of complex. Figure 3: Cost of Change Curves Cost of Change Boehm Cockburn/Ambler Beck Project Lifecycle Progress Object technology and modern integrated development environments (IDEs) boasting built-in testing and refactoring mechanisms negate the expensive Boehm cost of change curve and allow for the cheap change. an empirical or adaptive management approach is employed to measure and adjust the chemical process periodically to achieve the desired outcome. Although it has been attempted in the past. project plans are continuously inspected and adapted based on the empirical reality of the project. Agile Project Management: Empirical Process Scrum. In Copyright © 2004 Danube Technologies. . All rights reserved. Inc. there cannot be a single exhaustive library of defined processes to handle every situation that could possibly surface during a software project. a popular Agile project management method. Instead.

§ Requirements – The scope of the work that needs to be done can be increased or decreased to affect the project. The frequent demonstration and release of software common in Agile approaches Copyright © 2004 Danube Technologies. . Lean Thinking Another effective way to analyze how Agile methods increase efficiencies is to apply lean manufacturing principles to software development. Inc. In reality. Agile methods have their conceptual roots in the Japanese manufacturing productivity boom of the 1980s 19. Consider for example the “small batch” principle: things produced in smaller batches are of higher quality and efficiency because the feedback loop is short. Third. organizations can reduce risks by leaving options open to decide at a better time when more accurate information is available. and the development process dictates the fourth. when one changes at least one other factor must also change. It is therefore unreasonable to assume that management can control all four of these factors.18 The highly complex and uncertain nature of software development makes this expectation of full control unrealistic. Agile values a high visibility and customer involvement. Page 7 software development there are four broad control factors. controls can be adjusted more frequently. § Cost – or Effort. Agile methods encourage delaying irreversible decisions until the last responsible moment.17 Because software development is often considered a sequential. management can pick values for three of the four factors at most. Customers may not want to make decisions about the system until they have more information. § Schedule – A software project is impacted as the timeline is changed. All rights reserved. Available money impacts the amount of effort put into the system. thereby reducing the risks associated with large integrations at the tail end of projects. Many software development organizations that implement Agile software development are finding they get something they never expected: options. Rather than locking into decisions at the beginning of a project. Teams can work independently for a while but the code base never diverges for long periods of time. the concept of frequent or continuous integration keep software development teams synchronized.20 Agile Requirements: A Focus on Business ROI Agile projects avoid “up-front” requirements gathering for the reasons stated above: customers cannot effectively produce all requirements in high enough detail for implementation to occur at the beginning of a project. Although cross-industry analysis can be tenuous. § Quality – Cut corners by reducing quality. Second. However software development cannot be described by a simple linear process because it cannot be predicted accurately in advance. These factors are interconnected. middle and upper management often assumes that all four of these factors could be dictated to the development team under the waterfall approach. and resources are utilized efficiently to avoid “queuing” (see “queuing theory” and the theory of constraints). linear process.

Agile processes like Scrum and XP use a concept called velocity. This is especially the case with small businesses where domain experts wear many hats and often cannot commit to two or three months of straight requirements analysis. there is a valuable opportunity for the customer to re-examine business factors at the beginning of each iteration to select features for inclusion in the current iteration according to business ROI. The trendline can then be Copyright © 2004 Danube Technologies. Many organizations are not fully staffed with business analysts cranking out reams of requirements specs. Because we can change direction rapidly (every iteration) and the cost of change is low. “how can we budget for such a project?” It sounds a bit scary: let’s start working in short iterative cycles that yield demonstrable software without actually planning everything in advance. An iterative approach allows customers to delay decisions as well. a trendline can be established through the points to create a velocity (work the team can complete per iteration). the development team must bring technical risks to the customer. a Project Burndown Chart can be utilized to estimate the eventual conclusion of an estimated backlog of work. IID is ideally suited then to take on bite-sized chunks of requirements that the customer can easily digest. Of course. Convergence with Agile One of the most commonly asked questions by those examining Agile is. rather than trying to guess how things might shape up a priori. Agile helps companies produce the “right product”. Decisions can be delayed to some future iteration when better information or technology is available to optimize the choice. Focusing on high business value features first is therefore a critical component of efficient Agile development. “how do you know when the software will be finished if there’s no up-front plan?” and the obligatory follow up question. Quite the contrary. Each point of the chart in Figure 4 represents an iteration (or Sprint in Scrum). But we already know that we cannot plan for everything in advance. One of the biggest advantages to IID is that work can begin before all of the requirements are known. which is the amount of estimated effort a team can complete in a time-boxed iteration. The Agile answer is to examine project progress empirically. How do Agile projects prioritize work? A study by the Standish Group shows that in typical software systems 45 percent of the features are never actually used by users and another 19% are only rare used. As iterations progress. Once a team has established a velocity. We therefore built the system in a database independent manner. in our experience often the bottleneck in the development process has been the lack of availability of customer domain experts for detailed requirements analysis.21 This is largely because the unused features were specified in some up- front plan before the ratio of their cost to value was considered or even knowable. and the Y-axis represents the total estimated effort remaining for the backlog. and (luckily) a few weeks before the product launch a new version was released by one of the database vendors that solved our problem. Page 8 gives customers a chance to “try software” periodically and provide feedback. For example. . Inc. we recently delayed selecting a database package for an application because some of the desired features were not available at that time in the options we had to choose from. but in the end it is the customer that decides what the development team builds. All rights reserved.

Notice also that the velocity (slope) changed for the better. if the timeline and cost variables are fixed. To answer the questions above. which represents the empirical estimate of the completion date. . There is no guarantee all features will be built. For complex problems like project convergence. By iteration ten. Page 9 extrapolated to determine the X-intercept. All rights reserved. the product scope had been adjusted down so that the project could be completed by the budgeted 20th iteration. Inc. Agile methods tell us that the customer cannot specify all four of the software development variables (cost. adhering to strict code and testing standards. Copyright © 2004 Danube Technologies. Going to production with high priority features is better than never going to production at all. the project’s “burndown” trendline indicated an X-intercept well into the future (off the charts!). but it is certain that the highest priority features will go into production and that they will be built well. work will proceed on the highest priority requirements first to ensure that the most important things get done before a deadline or the money runs out. quality). the features built should always be high quality. especially considering the Standish report cited above that nearly 65% of features are never or rarely used in reality. And quality is non-negotiable. schedule. Through the first nine iterations.22 That is. scope. perhaps the reduction in scope was accompanied by the removal of critical impediments to efficient progress. Figure 4: Product Burndown Chart with Velocity 1200 Velocity: 5 story pts per sprint 1000 800 Story Points 600 Velocity: 25 story pts per sprint 400 200 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Sprints Figure 3 is a Product Burndown Chart representation of a typical project. In this case. then the scope of the work must be variable or the definition of the scope must be at a high level so the robustness of each feature can be negotiated.

It also nicely summarizes and contrasts some of the major Agile/Iterative approaches such as Scrum. software engineering has had a very short history relative to other engineering disciplines. Ken Schwaber. and the Unified Process. a highly effective agile project management method. but how do you do agile? Scrum is a very good place to start on the management side. In 1994. 23 Standish Chairman Jim Johnson attributes the improvement to smaller projects using iterative processes as opposed to the waterfall method. “Agile Software Development with Scrum”. and the most recent 2004 Standish Group CHAOS report on the success of software projects shows a dramatic improvement in the failure rate of software projects. Unlike bridge building. Craig Larman. 2001. Addison-Wesley. Theory and rhetoric are nice. software development is not built on thousands of years of trial and error. Inc. So what do your developers do differently in agile? Extreme Programming (or XP) advocates engineering principles such as pair programming and test driven development and Beck’s book is the de facto authority. 2003. Mary Poppendieck. Copyright © 2004 Danube Technologies. This book is outstanding and each page seems to offer a valuable nugget of information. much of this article leans on the Poppendieck’s work. Below are a few good starting points for anyone interested in learning more about agile software development. Agile is simply the latest theory that is widely replacing the waterfall approach that itself will change and evolve well into the future. It has the most comprehensive empirical evidence for Agile/Iterative of any book currently on the market. XP. . Addison-Wesley. Prentice Hall.24 The notion that Agile is a radical deviation from the long established. It contains the most compelling case for agile over sequential development I have yet to uncover. Mike Beedle. Tom Poppendieck.25 References After reading an introductory article. This book should be read by any manager interested in Agile. Page 10 Conclusion But does Agile/IID work? Of course the proof is always in the pudding. tried and true history of waterfall software development is incorrect. 2003. “Agile & Iterative Development: A Manager’s Guide”. Although waterfall is often referred to as “traditional”. sometimes I wish I could ask the author for a list of good books or articles to further my knowledge on the subject. “Extreme Programming Explained: Embrace Change”. Addison-Wesley. Kent Beck. Standish reported a 31% failure rate that has improved to 15% in 2004. and is therefore in a rapidly evolving infancy as an engineering discipline. This book is about “Scrum”. “Lean Software Development An Agile Toolkit”. 2000. All rights reserved. It also delivers outstanding tools for implementing agile.

http://www1. IEEE CS Press. 2004. Tom Poppendieck. Addison-Wesley. Addison-Wesley. 4 Ken Schwaber. 25 Software architect Michael James has a semi-serious theory that “test-only” development will soon be feasible with the advances in cheap. 15-19.com/articleshttps://www.scribd.com/designDead. 12 Kent Beck. Takeuchi. Nonaka. 100-101 17 In fact. 2000. software developers find it very objectionable when asked to skimp on quality. Inc. 137-146. Agile Modeling Essays excerpted from the book “The Object Primer. “Agile Software Development with Scrum”. Harvard Business Review. “Extreme Programming Explained: Embrace Change”. 2004.php 3 Kent Beck. 16 Ken Schwaber. 2003. 2004. Tom Poppendieck. http://www1. see: Mary Poppendieck. “Lean Software Development An Agile Toolkit”. by Alistair Cockburn. “The New New Product Development Game”. “Lean Software Development An Agile Toolkit”.php 6 I. 24 “Standish: Project Success Rates Improved Over 10 Years”. Published Keynote Third International Conference on Extreme Programming. Developers write only robust tests and supercomputers will write and compile code until all tests pass. 9 I. pp. It’s Your Job!”. Software developers should always aim for high quality software. 15 Martin Fowler. Harvard Business Review. 10 The Agile Manifesto is online at http://www. pp. clustered supercomputing. pp. “Chaos Chronicles”. p 32. 2 Standish Group International. 2000. “Extreme Programming Explained: Embrace Change”. Surgeons or lawyers would be sued for malpractice. Tom Poppendieck. “Managing the Development of Large Software Systems”. .com//sample_research/chaos_1994_1. Inc. Proc. and for the same ethical implications software developers resent this charge. 2001. “Extreme Programming Explained: Embrace Change”. January 1986. by Scott Ambler. “Chaos Chronicals”. pp.softwaremag. 48. Addison-Wesley. 2000.cfm?Doc=newsletter/2004-01-15/Standish.: Agile Model-Driven Development with UML2”. by Scott Ambler. 13 Reexamining the Cost of Change Curve. http://www. “Agile Software Development with Scrum”.. 22 Mary Poppendieck. September 2000. IEEE CS Press. p. Westcon. “Chaos Chronicals”. Victor R. pp. 2003. year 2000. Page 11 1 Winston Royce. 137-146. “The New New Product Development Game”. 14 Examining the Cost of Change Curve. June 2004.org/ 11 Barry Boehm. 2001. “ROI. 23 Standish Group International. 2002. pp.. it is debatable whether Quality is really an adjustable factor. Mike Beedle. Prentice Hall PTR. H.standishgroup. Mike Beedle. 8 Craig Larman. Computer. Takeuchi. Software Magazine and Weisner Publishing. p 28. XP Magazine.standishgroup.html . Addison-Wesley. “Is Design Dead”. 18-19. Cambridge University Press. 89-94 5 Standish Group International. 19 The Scrum Agile method has its roots in the Nonaka-Takeuchi article referenced above. pp.com/L. As professionals. Basili. “Iterative and Incremental Development: A Brief History”. Addison-Wesley. http://martinfowler. 1994. period. 2003. 1981. January 1986. “Lean Software Development An Agile Toolkit”.agilemanifesto. Addison-Wesley. Prentice Hall.com//sample_research/chaos_1994_1.. 1994. 3rd ed. Nonaka. 1970. All rights reserved. 20 Mary Poppendieck. “Software Engineering Economics”. Prentice Hall. 328-339. 21 Jim Johnson. Copyright © 2004 Danube Technologies. Inc. 18 Kent Beck. Inc. H. 7 For more on how lean development influences agile software development.