Agile or waterfall?

This blog post was born from a talk I did in a small drupal meetup, where I was presenting Kino.dks Streamingguide.

In the following blog post, I will discuss agile and waterfall development methodologies and align them with my own experience, both from a developer, but also from a product owner and project manager's point of view.

Background

I have had the pleasure of trying out a wide range of different development processes during my professional work life. First and foremost I want to point out that different models work well for different projects with different people. I do not believe that we have a one-size-fits-em-all model. Apart from my own personal projects, I 'v recently "finished" a mid-size project with a truly agile development process. Before that, I was the Product Owner of a very big project, run by the traditional waterfall methodology.

Waterfall

I am sure you are already familiar with the waterfall model. If you know everything about it, you should skip to the next section. I will briefly outline the most important concepts before we move on.

The waterfall methodology is as the name prescribes a linear process, where you start with a well-documented project description and then flow through different steps before it's done.

(image is from Wikipedia)

The image lays out the process pretty well. The project will usually go through the following steps before its completed:

1. Analysis; The project and especially the end goal is well defined
2. Design; From the requirements, a design is made. (This is often an iterative process)
3. Development; The developers transform the design into code
4. Test; It is being tested and aligned with the requirements
5. Maintenance; Updates and changes can be made from here and on.

I often see the waterfall methodology used for fixed-price-project, where the goal is clear and the uncertainty is low. Some would argue that it's not well suited for complex projects, but that is not necessarily true. Even a complex project can easily be run like this, but it does raises the requirements and sets some high demands on the documentation and project descriptions.

One important takeaway from this is; When using the waterfall model, it is important that you know "everything" from the start and has everything laid out and described. It is not well suited for changes in the requirements IN the process, therefor they have to be determined beforehand.

Agile

The Agile methodology differs from waterfall by not being a linear process, but rather a continuous loop of iterations; define, build, and release steps. So the process contains smaller steps, where you keep building upon the project and expanding the functionality.

The agile methodology appeared in 2001 and consists of a range of principles to follow. You can read the manifesto and principles behind here.

One of the advantages of the agile development process is that it lets you change requirements or solutions in the process. You work in small sprints where you build on top of the current product. (It could be an initial MVP) Often it is paired with a scrum methodology, where the team works in sprints.

What to choose?

Well, there is no silver bullet. It all depends on the project, the team, and the culture. There are a few things to consider though.
This is both the fun and interesting part of this blog post.

End product

Let's discuss the end product first.

The waterfall model is more predictable. You will most likely get the product you initially wanted.
The developers will be able to plan out the entire project from start to end. They will be able to make a lot of decisions and agreements before they start coding. This might lead to a better product, in terms of code quality. I do often see that the price is being stomped so that the pressure rises during the last period of the project and the developers are forced to rush through some parts. This often ends up with a decrease in quality.

On the other hand, the agile model is not as predictable. It can be, but it's often not. Problems are solved as they arrive and decisions are made on the fly. It's the agile methodology's strengths and weakness. It makes room for changes, which in some cases makes perfect sense, but can also lead to a different end product.

Work before or during the project

The approach to a project is very different from the two methodologies. One of the biggest differences is the work of preparation that's done before the project starts and during the project.

The waterfall model requires you to have solid and detailed documentation before the developers start coding. So an end-to-end description, which usually starts with a workshop where the requirements and limits are determined. Then the design process starts and usually, the developers use a combination of the documentation and the design to plan and build the project.

Agile is different, while you do prepare beforehand, it does consist of an incremental and iterative process. A scope is of course required and usually also a design, but there's room for unanswered questions.
The way I usually tackle an agile project is that I describe the absolute minimum requirements. I do keep a log of descriptions of a desired feature and result as well, but I update and edit during the process. So the agile process starts with an MVP and then builds on top of it, through an iterative process, where features/requirements are determined between each sprint. And eventually, you will run out of money or time and the project is done.

This mustn't be an excuse for not being well prepared!

Client and stakeholders' role

The client or stakeholders' role can vary a lot as well. Depending on the company, project, methodology, and so on.

I've tried both the very traditional waterfall, where the project starts after the scope and requirements are determined and then I've gotten weekly or bi-weekly status updates. Usually, I follow the project on the sideline and watch the process and only participate if there is uncertainty or some things that need clarification.

For most of the agile projects I've been involved in, both as a developer, product owner, and project manager, the client has been a lot closer to the process.
Some projects that I 'v been involved in, followed true agile processes.
A backlog of tasks, sprints of 1-2-3 weeks, and daily stand-ups, where challenges and progress were discussed. In these projects, the client was (together with the rest of the team) responsible for managing the backlog and continually testing the product.

Economy and challenges

The economy around a project is always a challenge. Either you have to figure out or agree on a price before you start, otherwise, you see how far you can get with the allocated resources. Neither is perfect.

I've often dreamt of a more agile economic setup, where the budget came in rates and if you were able to present a valid business case, the budget would dynamically increase. Following the agile methodology.

What usually happens for projects with the waterfall methodology, is that you make some kind of fixed-price project, which leaves no flexibility for challenges, details, or quality (expensive) decisions. You have to make ends meet, where the project is realized with the allocated resources. By doing this a wide range of risks appear, since you ignore the fact that you might learn something in the process. Or even worse, something that you haven't thought of came off.

For agile projects, there will always be a risk that the project will run wild or go crazy. You can end up with no usable product and no more resources.

Conclusion

Over the years I've concluded that I rarely can see and cover everything in a project. I have never been involved in a project, where I was able to figure everything out before the project started.
There have always been surprises and decisions to be made, that I wouldn't have been able to do or know at the start of the project.

The agile process does help me deal with this, but it leaves me with another problem; the budget/resources. It could seem like I have no idea what I'm doing and I just go with random development and gut feeling. I would argue that this is not the case. Often when I find myself in the decision land, it tends to ignite my innovation. This is helpful both in terms of concrete solutions, but also regarding improvements, new business ideas, or features. I believe that there can be great value in the process and should be considered.

But how can the economy and budget department follow agile development? Can we somehow align the two?

At most jobs I 'v had early budgets or fixed project budgets. I hope to get the chance to try a different and more agile model, where the budgets are made either monthly or quarterly. To be more flexible and grab the opportunities when they arise, instead of waiting till next year :)

So what do I prefer? Well, it depends on the situation, the project, and the team. If it's a team I know and trust and the project is perfectly scoped and clearly defined, the waterfall methodology works great for me. If the project is less accurate or weaker defined and I need to utilize the knowledge of the team, in-depth, then I would go with the agile process.

Again, no simple answer, but I see great value in both. The conclusion is that both of them are great for different scenarios. Experience and understanding the strengths and weaknesses of the two is the key value.