Articles / Xtreme Programming and Open...

Xtreme Programming and Open Source Software Development

Over the (fairly) recent past, software developers have been offered a plethora of panacean remedies that seek to address inherent inadequacies and observed problems in traditional software development methodologies. More often than not, however, they come bundled with their own variants of the said inadequacies and problems, and (at least as far as software developers are concerned) basically replace an old devil with a new. But not all do this. One particular credo that has seen some well-deserved success in this regard is Xtreme Programming.

Xtreme Programming, with due credit to its founding fathers, bases its processes and practices on elements of simple and sound common sense. The focus in XP is on the things that matter in the development of software (a detail that seems sadly lacking in the exercise of prevailing methodologies). There are whole volumes available on XP that discuss what "really matters" in software development, and I won't dwell on it here because XP is not what this article is about. This article is about Open Source software development. Given the unrelenting ideal of better software that remains the guiding principle of the Open Source world, it is inevitable that it crosses paths with XP, in philosophy to a large extent and in practice to some. In the rest of this piece, I'll examine some "Xtreme" influences on Open Source software development -- what these are and what these can be.

The customer

XP recognizes that the single most important factor in software development is (or at least should be) the customer. Most key practices in XP revolve around communication with the customer, "listening" to his requirements and "responding" to his feedback. XP even advises that you have a representative of the customer available at the development site throughout the development lifecycle. Open Source projects have a quirky and largely unintentional similarity to this approach. Open Source projects, almost without exception, are born as tools to serve some personal need of the developer, who, eventually, comes to believe that the tool may benefit the rest of mankind, and publishes it. Others then try it, improve it, hack it, rewrite it, or supplement it to address some missing feature or to fix a bug. So, at least in the beginning stages of the development, the developers are the customers. And when the package is actually released to the community, rich feedback in the forms of reviews, bug reports, usage experiences, and comparisons, the like of which developers of proprietary software can only dream of, show up on the ubiquitous mailing lists. All in all, the customer factor in Open Source development is handled in a manner that even XP can learn from.

Releases

Most conventional software development methodologies work on a single release schedule. The development lifecycle spans years and is essentially blackbox from the point-of-view of the customer, who has to step out of the way and patiently await the day the developers transform his requirements into usable software. This approach offers no scope for change, for feedback from the customer on what is being churned out in the development process, or for a cut-and-run. For customers who need some degree of control on the development process, XP is often the answer. XP proposes an incremental release schedule that incorporates a finite subset of product features (sorted by business value) in each release. Customers feed requirement changes as well as their comments on the current release into a system that adopts these into the release cycle. And, importantly, each release is throughly tested, and the customer is assured of flawless execution within the scope of the limited feature set.

Iterative development of this sort is the norm in Open Source development. In fact, it's so much so that the Open Source SDLC has been referred to disparagingly as "release-first". To be fair, some of this criticism is deserved. Open Source package releases are often buggy and full of untested code, thanks to authors who have come to depend on bug reports and feedback from users to such an extent that they do not even perform elementary sanity checks on their code before publishing it. To be fair some more (to the developers this time), it must be mentioned that such releases are tagged appropriately and advertised as unstable. However, that's no excuse. One lesson that Open Source developers can take from XP here is to ensure that any release that they make is thoroughly tested in scenarios that the developers are familiar with. Feedback from the field can then be relied on for scenarios that are untestable in the development setup or which have been forgivably forgotten.

In the same context, XP assures its customers strict adherence to a pre-planned release schedule. Though the scope of each release may be varied (in consultation with the customer), the deadlines remain immovable. On the contrary, the release "schedule" in Open Source projects is, to say the least, haphazard. Planning is sometimes based on a TODO list, but is more often governed by essential bugfixes and the "triviality factor" of feature additions. Since most development endeavors in the Open Source world happen in the off hours when the developers are not at their regular day jobs, we sometimes see releases that follow each other with scarcely a day in between followed by lulls that may span years. Exceptions to this are the large projects with roadmaps that sketch the broad outline for a release schedule.

Formalism

The verbiage and unnecessary generalization of formal SDLC documents often end up cloaking the essentials of software development, what the customer wants his software to do (requirements) and how the developer satisfies these needs (design and implementation). When documentation (of this sort) becomes the mainstay of the development process, the only other supplicant in the process, the software itself, suffers. XP is decidedly against such formalism. XP uses user stories as alternatives to bulky software requirements specifications. These describe usage scenarios drafted by the customer, in his language and as he understands them. And instead of UML-based design documents, XP suggests simple CRC cards that capture the essence of the design in the developers' lexicon on a medium that is easy to retrieve and refer to.

The Open Source community, in its turn, never did believe in the idea of tracking the development process in formal documents. Design documents are usually written to serve as aids to future developers or maintainers. But the denizens of the Open Source world are well-versed in the art of browsing code to understand design and have never needed reams of paper describing it. How about a requirements document, then? Most projects are proposed as ideas on a mailing list. Developers thresh out the design through mail exchanges and posts. Eventually, a skeleton forms that is fleshed out in versions. Often, the bulwark of project documents is the set of user guides. In large projects with several modules and intricate cohesion, documents describing the interactions between the various modules and control flows are, however, essential. To their credit, Open Source developers have been quick to realize this. Most big Open Source development efforts today are backed with the necessary design descriptions.

KISS

Code the simplest thing that could possibly work. This simple idiom is another of XP's cornerstones. An ideal that could be one of XP's most valuable contributions to software development, this still remains one of the most difficult to adhere to. Most developers have a tendency, well-intentioned no doubt, to ensure right from the beginning that their code will cater to the wildest whims of 0.02% percent of their target users. Thanks to this, they end up cluttering their code with so many options that the usage info tends to span 6 screen pages. Or maybe they have chosen to incorporate into their code the best possible implementation of one of the more obscure components, thereby ensuring that the glue that attaches that component with the rest of the stuff is fragile. It is not the catering or the choice of the best implementation that XP looks askance at. It is the timing. It is not necessary that your first release be a study in optimization. Instead, let it be a study in simplicity, in functionality. Let it do all it set out to do and do it in the simplest manner possible. Let it give users a feel for what they can expect of their software. Chances are, they will be happy with the functionality. Optimization without user-perceptible benefits is one of the most unrewarding of activities in software development. Though this particular tenet is something that anybody, no matter how accomplished, can learn something from, Open Source practitioners are fairly adept at making this work. The manner in which they go about doing this, however, may leave something to be desired. Most initial releases don't even do half of what they are supposed to, leaving subsequent releases to implement the missing functionality. Substantially down the version chain, one finds releases that focus on optimization. Though admittedly a corruption of the concept, this does help avoid the pitfalls discussed above.

Refactoring

Refactoring is not easy. It requires the developer to fight one of his greatest fears, that of change, and indulge in an activity that would cause apoplexies over at "upper management" -- complete and merciless redesign of "perfectly good code". It requires that he have enough objectivity to perceive that what was once a beautiful piece of code or design is now archaic, bloated, or simply bad. It requires an understanding that times change and software must keep pace. An understanding that holding on to your brainchild is a sure way to stunt its growth and deny it opportunity to mature and evolve. Open Source developers are getting better at this. And "better", when you consider that the concept has yet to sink in with the proprietary software crowd, is actually pretty good. The Open Source world has long been the breeding ground of hacks and patches. But eventually, there comes a time when developers (or maintainers) grow weary of code that is so chock-a-block full of hacks and workarounds that even new features are implemented as hacks. This is when they bring out not their customary erasers, but a fresh whiteboard. This is when they sit down to rewriting their code, incorporating changes and assimilating workarounds and hacks right into the design, with little regard for the previous work. Didn't we coin the term "from-scratch rewrite"?

This-is-your-code

In the archetypal software development scenario, the software system is split into a set of modules (sub-systems), each of which is assigned to an individual or team. Though this divide-and-conquer strategy works well in practice, it has the inherent shortcoming of insulation. The teams remain isolated in their domains of expertise with regards to the system, and nobody from a team working on module A can even venture an answer to a query on module B, far from being qualified to answer it. XP proposes the strategy of Collective Code Ownership to tackle this fly in the otherwise effective ointment of module-based development. Collective Code Ownership makes every developer working on a project responsible for all the code in the project. Anybody can propose and make changes to any section of the code or design. The doyens of distributed development, the savants of version control, and patch gurus, Open Source developers are completely at home with this strategy. This is understandable because they had this one nailed long before it became fashionable. Most development in the Open Source world happens across continents, across timezones, and often across projects. Nearly everyone who uses the code is eligible to participate in its evolution. He can contribute to the design, write code, fix bugs, document, and influence the contents of the next release. This open-arms, this-is-your-software attitude towards users that is typical of Open Source software development is, if you come to think of it, its very foundation.

A related notion is that of continuous integration. Changes made to a shared codebase by multiple developers are merged, and conflicts (if any) are resolved. Regression tests are conducted to ensure that changes introduced by one developer do not corrupt the working of another's. If all is well, these changes are assimilated into the release cycle. A natural fallout of community development, this is handled effortlessly in the Open Source world. Patches may be submitted for consideration by all and sundry to the maintainers and primary developers. If a patch is adjudged worthy of integration (criteria for which is advertised well in advance), tests are run to verify that it can coexist with other accepted patches and will not break anything in the current codebase. Following this, it is incorporated into the base.

System Metaphor

A system metaphor is essential when a wide cross-section of developers are working on a shared code base. It brings consistency in nomenclature and language so that anyone who looks at a section of code understands what the code is basically about, even if he doesn't understand it all. This ensures that all developers are on the same page with regard to the system, and encourages collective code ownership. Coding standards of this sort are practically non-existent in the Open Source world. Developers tend to use their own vocabularies and their own ideas of how code ought to be written, effectively rendering their produce incomprehensible to most other developers. Of course, it must be noted here that Open Source developers are remarkably proficient at understanding code written by someone else, so lack of system metaphors is not (generally) a significant problem. Still, coding standards are being developed and advocated by several organizations and forums now. These, if adopted, will only serve to enhance the productivity of the Open Source community.

Other ideas

There are other maxims in XP that have yet to make an appearance in the Open Source world. These include interesting techniques such as pair programming and test-driven development. Though their usefulness and general good sense cannot be disputed, the fact that they require rather sharp breaks from what developers identify as "normal" development is to blame for their constrained adoption when compared to other key practices in XP.

Conclusion

Given that this article only sought to examine and comment on (rather than judge) influences of Xtreme Programming on Open Source software development, this conclusion (unlike the rest of the article) is going to be brief. We have seen substantial evidence of the "Xtreme" in Open Source development. And though I call them "Xtreme influences", I hasten to add that it is considerably more probable that XP, in its assimilation of best practices from other methodologies and its innate focus on common sense, adopted some of these tenets from the Open Source world. Whatever or however unintentional the influence, I propose that it has all been to the good and must only grow. Xtreme Programming is practical. Its tenets are infinitely applicable and designed for present-day software development. Open Source is mature and adaptable, and is, quite simply, the future. A confluence is welcome.

RSS Recent comments

01 May 2004 06:16 mgruenke

XP & Open Sounce Development share the same roots
To me, it seems that both XP and most Open Source are rooted in minimization of planning, organization, testing, etc. (i.e. all non-programming tasks). Rather, both tend to focus on maximizing time spent programming. While I think Open Source developers can hardly be blamed for this, it seems to me this aspect of XP is the main reason it's so popular with programmers (who are notoriously lazy about non-programming tasks).

BTW, I think there are some mis-characterizations of proprietary software development in this piece. For example, all commercial software development projects in which I have been involved have engaged customers very early in the development process and done alpha and beta releases as soon as there's an interesting degree of functionality that would make it worth the customer's time to use. At my current job, we even do things like getting customer feedback on UI mockups. It's my understanding that these are all relatively standard practices.

01 May 2004 14:51 egorFiNE

Re: XP & Open Sounce Development share the same roots

> BTW, I think there are some

> mis-characterizations of proprietary

> software development in this piece. For

> example, all commercial software

> development projects in which I have

> been involved have engaged customers

> very early in the development process

> and done alpha and beta releases as soon

> as there's an interesting degree of

> functionality that would make it worth

> the customer's time to use. At my

> current job, we even do things like

> getting customer feedback on UI mockups.

Hmm, my experience shows that this may sometimes be too expensive for development companies. Say, it's the case when the customer is ignorant and intermediate releases makes the customer disappointed and developers crazy of really stupid questions.

In Russia there is a famous saying: "never show halfdone work to an idiot".

How do you handle these situations?

01 May 2004 17:37 Avatar tthomas48

Re: XP & Open Sounce Development share the same roots

%While I think

> Open Source developers can hardly be

> blamed for this, it seems to me this

> aspect of XP is the main reason it's so

> popular with programmers (who are

> notoriously lazy about non-programming

> tasks).

I think this trivializes a bit. Much of extreme programming actually starts with building tests, which is something most programmers hate doing.

You say programmers are notoriously lazy about non-programming tasks, but I haven't seen the QA team fixing problems in their environment, or upper management install their own copies of Microsoft Office lately either.

I think that XP is actually about the inverse. It's about not building the super-amazing-stupendous version of every product. The perfect product that can not be delivered. It's about building something releastic that can be delivered on time, while still putting a rational amount of thought into testing and future extensibility.

02 May 2004 02:04 S9

Re: XP & Open Sounce Development share the same roots

> Hmm, my experience shows that this may

> sometimes be too expensive for

> development companies. Say, it's the

> case when the customer is ignorant and

> intermediate releases makes the customer

> disappointed and developers crazy of

> really stupid questions.

>

> In Russia there is a famous saying:

> "never show halfdone work to an

> idiot".

>

> How do you handle these situations?

This is impossible in XP because of two things:

In XP there is no such thing as a "half done work." The idea is that the product should be at any given time in a state that it can be shipped and it will be valuable to the customer (well, maybe not during the first few weeks, but that's not different from any other processes).

Also in XP the customer is constantly following the development and giving feedback. So she constantly sees how the software evolves and does not think a certain intermediate release is the final result.

02 May 2004 10:50 vinodv

Re: XP & Open Sounce Development share the same roots

> BTW, I think there are some

> mis-characterizations of proprietary

> software development in this piece. For

> example, all commercial software

> development projects in which I have

> been involved have engaged customers

> very early in the development process

> and done alpha and beta releases as soon

> as there's an interesting degree of

> functionality that would make it worth

> the customer's time to use. At my

> current job, we even do things like

> getting customer feedback on UI mockups.

Matt,

Perhaps I have been guilty of some unwarranted generalization. But hardly, I think, of mis-characterization. For one, the section you quote compares XP with 'conventional' software development methodologies, not 'proprietary' software development (whatever that is). And a comparison of these traditional development methodologies with more agile ones like XP, clears shows that XP offers customers a wider, more influential role in the _entire_ development lifecycle (not just validation, which is plenty much what your examples describe).

Further, if you have developed in scenarios where customers were engaged early and _remained_ engaged throughout, you probably were following some tenets of Agile development techniques in your process

> It's my understanding that these are

> all relatively standard practices.

Again, perhaps. But like I said, the kind of practices you describe were not the kind that particular section was about.

02 May 2004 21:10 togaman

Re: XP & Open Sounce Development share the same roots

> BTW, I think there are some

> mis-characterizations of proprietary

> software development in this piece.

I see some generalizations on all sides but on the whole pretty balanced. My experience of writing proprietary software for internal customers in a Fortune x00 company has been fascinating - I began doing XP 18 months ago and it prompted me to take my Open Source project offline until a full test-suite had been written. The process of applying XP principles to my Open Source code made it all the stronger improving the design greatly. Everything I learned about XP I got doing test-driven development, writing proprietary software.

In other words, a powerful cross-fertilization. Good writing, keep up the good work!

05 May 2004 05:05 egorFiNE

Re: XP & Open Sounce Development share the same roots

> This is impossible in XP because of two

> things:

>

> In XP there is no such thing as a "half

> done work." The idea is that the product

> should be at any given time in a state

> that it can be shipped and it will be

> valuable to the customer (well, maybe

> not during the first few weeks, but

> that's not different from any other

> processes).

You mean that until the first build is done you show nothing to your customers? So the first version shown already has something useful in it?

06 May 2004 02:15 S9

Re: XP & Open Sounce Development share the same roots

> You mean that until the first build is

> done you show nothing to your customers?

> So the first version shown already has

> something useful in it?

Don't be stupid.

21 Aug 2004 19:57 antrik

Some criticism
This is a really interesting, and for the most part very accurate analysis.

There are a few points I don't agree on, however.

Releases (testing): My believe is that the extremely high productivity of the

developers in popular free software projects is owed mostly to the fact that

they do not have to bother with stupid side tasks like testing. Trying to

change this probably would be highly counterproductive.

This doesn't mean some automated tests aren't useful, though -- especially for

all those less known projects with a smaller user base generating timely

feedback...

KISS: While some free software projects are really dirty hacks, the greater

part actually suffers from massive overgeneralization. Many projects spend more

time writing insanely generic libraries or components (that noone else will

ever use), than on developing the actual application.

Refactoring: Here I have to protest violently. Refactoring is actually a thing

free software developers are *extremly* bad at. I have a hard time to think of

any free software project that does any serious refactoring at all.

Usually we just hack and rehack the code, until someone states that it's ugly

and should be thrown overboard. So the developers dump the code that has

matured over years, and instead sit down to write from scratch. After a long

long time (usually several years) -- during which hardly any real improvment to

the software take place -- they finally come up with something that is

completly overgeneralized, bloated, slowass, and terribly buggy. The following

years are spent on regaining at least part of the performance, stability and

functionality of the old code. (Just think of the KDE2 disaster.)

This is exactly the *opposite* of refactoring! Refactoring is all about

*gradual* cleanups/design improvements, about never letting the code get stale,

in short: About never throwing away larger amounts of good, matured code. Of

any lessons free software developers could learn from modern software

developement methologies, this is *by far* the most important one!

20 Sep 2004 06:36 mdnava

Keeping things simpler but well done
I think anyways that our coding should be very well formatted and documented but this XP seems far too expensive for small to mid projects.

Also I gotta say, the XP website (www.extremeprogramming...) needs better organization. I was reading for ten minutes and I didn't find a practical example (not saying there isn't one)

I prefer good object oriented programming rather than extreme programming but it looks interesting. Still, I'd like to see practical applications.

Screenshot

Project Spotlight

Bible-Discovery

Bible study and concordance software.

Screenshot

Project Spotlight

Possum

A point of sales system for untamed managers.