Recently, Forbes posted an article titled “The End of Agile”. Of course, this article caused the expected tweetstorms with people arguing whether agile has already died or if it’s on its last legs. Interestingly enough, a colleague shared the article in our internal chat and a very lively conversation followed, which led me to do some more thinking during the days that followed. Finally, I got to go on vacations for 10 days. Of course, as a software developer that respects oneself, my first action item for my holidays was sit down and write a blog post on these thoughts. But, I also had to sugarcoat it with a clickbait title first. Judging from your click-through, I didn’t do too badly!

In this blog post, I will go over some of the arguments laid down in the article and some complaints about agile and the various methodologies expressed by various colleagues, friends and other people across the Internet. I want to emphasize that in no way this means I am critical against these opinions. They are all formed by people working in the industry for many decades, thus carrying useful insight from their past experiences.

Let’s start first with some points raised in the article:

• (1) Some agile methodologies include daily standups, where the members of a team are sharing what they have been doing with the goal of providing visibility and helping each other when they are blocked. The article describes a degenerate case of this process that can happen in organisations, where this meeting is used purely as a reporting mechanism by management, instead of being there to serve the members of the team.
• (2) Creating software is not predictable. As a consequence, it is not measurable and it’s hard to provide estimates for software development tasks, especially ones where nothing similar has been done in the past by the team.
• (3) There is a fallacy in computer programming circles that all applications are ultimately decomposable. Furthermore, most of the effort sometimes comes from integrating the various components rather than implementing the component themselves. The end result is that problems around availability, resource usage, synchronisation, race conditions and performance tend to surface only in the end after all the integration is complete. This is one of the main reasons developers are reluctant to provide estimates for their work. This becomes even more of an issue when developers have to integrate their efforts with other developers, especially for those components developed at the same time. An impedance mismatch in the interaction between components can take significant amount of time to resolve.
• (4) The constant emphasis on minimum viable product means that at any given point developers end up spending a significant percentage of their time building and demonstrating their work to date, eating up another ten to fifteen percent of their available time, often for code that will be thrown away.

Now, let’s see some complaints expressed from friends and colleagues:

• (5) It is difficult to separate the concept of agile from implementations and implementations end up being process heavy (i.e. lots of ceremonies & meetings).
• (6) A lot of organisations do not allow too much customisation of processes, e.g. according to an anecdote, teams were not allowed to use a different methodology instead of Scrum.
• (7) If agile assumes anything, it’s changing software is easy. This can be true when users of the software are humans (i.e. web interfaces). However, it’s harder when developing an API, at least one with a large surface area.
• (8) Agile suffers from the “No true Scotsman” fallacy. When people criticise some part of a methodology, agile proponents argue that this specific element is not really “agile”, thus making it really hard to figure out what agile really is.
• (9) Agile provides space for a lot of unproductive vagueness given the customer provides continuous, iterative feedback which can be contradictive at times. This is in contrast to waterfall approaches that were putting focus on establish a contract via requirements analysis, which makes it easier to resolve disputes where the customer isn’t consistent about what they ask.
• (10) Agile forces splitting up work that shouldn’t be split up. As a result, developers lose the big picture.
• (11) Agile assumes that developers are replaceable and interchangeable. Anybody can do any work as long as the JIRA ticket has enough detail and everybody will be equally efficient at it.
• (12) Some of the ideas behind the agile manifesto were put down by people that observed mature and experienced teams getting lots of stuff delivered and wrote down what they saw. However, they missed all the parts where these experienced developers optimised away unnecessary conversations and design discussions, because they all knew each other and had lots of domain knowledge. These crucial, human interactions are essentially omitted in the various methodologies and their associated tracking tools.

Before going through some of these points, I think there is an important distinction to be made - the difference between principles and processes. Once upon a time during my undergraduate studies, the professor of our operating systems class was emphasizing the difference between mechanisms and policies almost every single day. That seemed eccentric back then, but I can now see the real value of this distinction. To get back on topic, principles essentially define how we think about what we do, while processes determine what we actually do. In most of the cases, the former gives shape to the latter. However, people sometimes can get obsessed with the processes in place and forget the overarching principles that led to these processes. It’s important to take a step back, re-visit the principles and see if the processes have to be adjusted accordingly if they don’t serve us anymore.

So, let’s look at the 4 major principles of the agile manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

An interesting contrast emerges now. Let’s go and have a look at all the points raised previously:

• (1), (2), (5), (6), (8), (10), (11) and (12) are all cases where people optimised processes and tools over individuals and collaborations - so one could say not really agile. Let’s look at them separately.
  • (1) is most likely a case of Scrum gone wrong and people misunderstanding the purpose of some of the processes.
  • (5), (6) is an indication of lack of retrospection and adaptation of processes depending on what works and what doesn’t.
  • Unfortunately, (8) is an indication of the obsession of people on processes and tools. With the proliferation of methodologies and the corresponding certifications, people started putting two much emphasis on the processes used, instead of understanding what’s the value of these processes and why they are encouraged. We finally ended up arguing whether we are doing agile (whatever that means), instead of trying to understand whether we are as efficient as possible and how we can improve.
  • (10) is an interesting one. I am pretty sure you have been in a room, where a story is too big and needs to be broken down into smaller stories in order to do “proper Scrum”. At the same time, most probably you’ve also dealt with tasks that might have involved some design or exploration aspect that had required a significant amount of time to complete, while also containing a lot of ambiguity. I believe there is some truth in this point, but I also believe there are cases where it’s just a symptom of an underlying problem. It is easy for someone to confuse whether they are doing exploration or actual implementation. I have been in this place myself in the past. I was exploring a new testing framework for one of our projects and instead of only checking whether it’s feasible and what would be the biggest pieces of work, I was spending time on implementation aspects that we could flesh out later on. So, it’s important to differentiate between exploration, design and implementation work and act accordingly. This is also a way that can help in doing rough estimation on tasks that are breaking fresh ground and we have limited understanding (see point (2)). By the way, some methodologies already have a name for the concept of timeboxed exploration tasks, called a spike. By splitting work into exploration, design and implementation phases, it is easier to keep track of progress. It’s also easier to determine whether something is done, since each phase can have different acceptance criteria. For example, the goal of the exploration phase might be to just identify whether something is feasible, the goal of the design phase might be to identify how something will integrate with other parts and the implementation phase needs to ensure any new code is tested properly and has the proper quality. Moreover, I believe splitting tasks into smaller sub-tasks does not necessarily force developers to lose sight of the big picture. On the contrary, I have been in teams where engineers would own the design of features and would be responsible for driving the team through the design and the various items of work, so that all the members of the team would be able to give feedback on the design and contribute to the implementation as well. I am sure this provides a lot more visibility to developers when compared to a single developer developing a single feature on his/her own with the rest members of the team having little understanding of it. What’s worse, the effects of this phenomenon are compounding later on when people have to work on or integrate with parts of the systems they haven’t worked on at all. Again, individuals and interactions over processes and tools!
  • (11) is also a quite problematic situation. I think it mostly occurs in situations where estimates are used as a management & reporting scheme, instead of a somewhat speculative planning scheme. On the other hand, having silos of knowledge in a development team is also an antipattern, as described previously. So, having huge variance in the time required to complete something between two different members of a team is usually not a healthy signal. What is also important is the scope of the associated work. For instance, domain knowledge of individuals can be much more impactful in earlier, design phases when compared to later implementation phases, where members of the team should have a shared understanding of what needs to be done. This is not to say that we should encourage treating software engineers as fungible resources. Instead, it’s meant to emphasise that through teamwork and collaboration we can achieve more than we could achieve as individuals. It is also easier to derive value from peer review and feedback in this way, since knowledge is shared and there are fewer silos.
  • (12) says it all, I think. There is nothing more to add, besides repeating the phrase of the first principle - Individuals and interactions over processes and tools.
• (3) is one of the ugly truths of software development. It’s hard and complex and there are many unknowns that need to be discovered one at a time. However, doing big-bang releases and big design up front (BDUF) usually makes things worse. Decomposition can work in multiple dimensions. Instead of starting from the small components and leaving all the integration for the end, a team can work in vertical slices interleaving implementation with integration work. Agile thinking can help in this by encouraging people to think which pieces introduce most of the risk, so that work can be planned accordingly. Also, collaboration is one of the basic agile principles and it’s how teams can reduce this risk to some extent. Instead of trying to design in isolation, teams should engage each other, combine their domain knowledge and think collectively how the integration should work.
• (9) is a valid point that can be challenging for many businesses. I personally believe it’s still mostly a cultural problem, not a technical or mechanical one. As quoted previously, agile favors customer collaboration over contract negotiation. The unfortunate truth is that even the customer might not really know what they want and the only way to find that is through collaborative exploration. Following a waterfall approach, one can sit down with the customer and decide on a set of a requirements and a long-term plan on how to achieve them. However, when following agile thinking, one will also try to see how this plan can be split in smaller milestones, where validation and calibration can be done when needed. In this way, there is less wasted effort and more productivity on both sides of the equation. To achieve that though, a shift in mindset is needed not only on the technical side, but also on the side of the customer that must acknowledge they might not really know what they need and it’s a team effort to figure that out incrementally.
• (4) is also a valid complaint. However, no one talks about the value gained from these sessions through the provided feedback and the knowledge sharing across the team. This is an omnipresent trade-off in the software field. I think there is actually a spectrum. The one end is working fully in isolation and reaching full individual productivity to the detriment of potentially wasted effort and other members of the team left behind. The other end of the spectrum is overabundance of meetings, tech talks and demos where people get limited time to do actual development work. Common sense is the answer to find the right balance in this. Scrum, Kanban et al can’t do much to help you on this really.
• (7) is a very interesting point. Different forms of software are subject to different constraints. For instance, evolving a platform API can be quite challenging, since one has to take a lot of different aspects into consideration, such as maintaining backwards compatibility, interaction between conflicting features etc. I believe agile thinking can still be helpful here by providing a mental framework of thinking how changes can be made smaller and how validation and feedback loops can be introduced in the various stages of development to make these changes safer and easier to complete (e.g. via automated testing).

I believe all of the complaints and arguments raised above are very useful, since they come from practitioners and reflect how these principles play in the wild. I also believe discussing them openly and trying to find the weak spots of current practices is the only way to improve our craft and become better in what we do.

And remember. It’s not the end, it’s just the beginning..