Fourth Myth of Agile

Quality is lost with Agile.

The Fourth Agile Myth

What is Quality …?

If we can be confident that our product will fulfil its primary purpose consistently and reliably, then it will be a high quality product. Furthermore, we can be confident in the product because we can prove its performance.

There are many indicators that can illustrate the quality of a given product – whether that product is purely software based or otherwise. For example, a low quality product may be hard to use, unreliable, incompatible, expensive, slow or even dangerous. We try to measure these aspects of our product, so we can make sure that the user is satisfied.

Bugs

A bug represents some situation where the product is not behaving as intended. Unsurprisingly, the premier indication of quality in a software system is the bug count.

A high incidence of bugs usually means we’re making a low quality product. To resolve a bug, we need to spend time and money, so a lot of bugs means that low quality can cost us a fortune. Frankly, it means we don’t really know what we’re doing, and whatever it is, we might be doing it in front of our users with our potentially shippable product.

Worryingly, a low bug count doesn’t necessarily mean that our quality is high. It could also mean that we’re not testing our product. Likewise, if more bugs are being reported by users than our own testing activity, then we’re creating a lot of bugs we can’t even detect.

Time and Money

Time and money are two things that our stakeholders don’t want to be spending. It takes a lot of time and money to develop a viable product, and this will be compounded by a lot of bugs.

Moreover, even without the bugs, if it takes a long time to develop the product then we might be developing it in a low quality way.

Rapid Development

Agile techniques encourage us to generate code that gives the user something to use early on. We then improve our code based on their feedback.

Feedback may include new or refined features, but it could also be plain old bugs. The crudest development strategy keeps us developing the code until the issues subside, like an outgoing wave on the shore.

When the issues subside, we think we’ve made something that the users want. We were Agile, because we engaged with the users (tick) and we delivered little and often (tick).

This is often a perfectly viable approach, as long as our stakeholders are willing to keep on spending, as I explained in the first Agile myth. But it’s a very reactive way of developing code, and we might have been able to avoid some of the refactoring and many of the bugs if we approached things differently.

Bugs are inherent to software; but it would be better to have fewer of them. Refactoring is great; but it would be better if we didn’t have to do it at all …

Simplicity – the art of maximising the amount of work not done – is essential.

Agile Manifesto

Each change in the code introduces the risk of more bugs. Rash (or “small”) changes to code make this risk even bigger, often for the reasons I explained in the third Agile myth. Lots of short-sighted small changes often make the code less stable. Therefore, a reactive approach to Agile development can easily damage our quality.

Agile is not a development process. It’s not the Agile manifesto that defines the quality of our software.

The software development process defines the quality of the software.

Agile Helpie

Proactive Agile

Agile concepts are intended to help us deliver working code. The code ultimately provides the functionality that the users want or need. In many ways, if the code is being produced on time and all the stakeholders are truly satisfied, then there’s no problem. In that instance, you could say there’s no point reading the rest of this article …

Unfortunately, the utopia of on-time bug-free full-functionality is harder to reach for some domains than others.

Small start-ups that build their products on firm IoT foundations and web frameworks have a head start. Often, they can provide valuable services by focusing on the business logic and not building the platforms upon which they need to sit. So wherever possible, it really helps to define products (or at least aspects of them) that can exploit these foundations. That’s a fundamental concept for any viable product architecture.

But when it comes to developing products for niche, uncharted or high-integrity domains like space exploration or medical equipment, a large chunk of those important foundations need to be dug out by the development team.

In these domains, the end users simply can’t afford to meander through endless waves of issues as they are discovered. Performance and stability must be absolute, the budget is finite and the timescale is constrained.

In these circumstances, we need a more proactive way of exploiting Agile techniques to deliver robust products predictably. If we can achieve that with complex, high-integrity products, then why not use the same techniques to develop any product more efficiently?

Just Enough Process

Many developers baulk at the complex bureaucracy of traditional quality processes when developing code. The tedium of document reviews and the controversy of coding standards take the fun out of crafting something new.

But it’s naive to think that well-formed processes hinder development. When deployed appropriately, such processes can be proven to increase the product’s quality and the team’s efficiency. It’s only when processes are deployed inappropriately that they stifle creativity and delay delivery.

The Agile manifesto assumes that a team of competent developers knows exactly which standards and processes they should apply to code written within their domain – but at no point does it state that there should be no process at all. The manifesto simply favours techniques that make people work together.

For example, working together as team means that we are always reviewing our work. The theory behind Extreme Programming means that we combine design, testing and code – and we review all of those activities in-line.

The hallowed pillars of Scrum actually lead to a highly regulated framework of meetings and methods – and it simply won’t provide any benefit if we don’t buy in to them. These processes are therefore some of the strictest around.

We need to define just enough of the right development processes to be confident we can deliver the right product on time. So how do we exploit Agile techniques in a proactive way?

We define a process that delivers the right code first time.

That is (literally) a bold statement. And it’s easier said than done. Most development processes aren’t portable across domains and organisations; but the fundamental concepts can still be applied.

In terms of process, Scrum gives us critical reference points that manage software development, but it doesn’t tell us precisely how to create the right code. It assumes we can do that bit, because we’re professional developers and we know what we’re doing.

We can start with two key processes. Indeed, we might need more than two processes to meet the ultimate gold-standard in certified level 5 capability; but these two are particularly important:

1. We need to go through some process to define tasks that we know we can complete within a defined period.
2. We need to go through some process that helps us deliver the code with minimal bugs and minimal refactoring downstream.

Load the Product Backlog

The product backlog contains all the tasks we know of that will fulfil the users’ needs. Scrum and Agile bang on about adding user value with every sprint. But how do we translate the needs of the users into tasks that will both add genuine value to the product and also be completed within a defined period of time? Moreover, how can we ensure that we minimise the expenses incurred by bugs and refactoring? Surely the management need to supply us with some kind of magic bean or silver bullet?

The key to loading the backlog is high-level design. We need to do just enough high level design up front to identify the items we need to build, how they interact and the order they should be developed. Therefore, our process to load the backlog includes this high level design activity.

High level design must translate broad user desires into a solution that we can build up over time. Our design activity allows us to divide the required functionality across notional components, and define how those components interact using interfaces.

Naive developers often eschew the concept of design, declaring that “Agile says documentation is a waste of time”. In fact, design activity is not equivalent to a document; and furthermore Agile does not say that documentation is a waste of time:

We favour working software over comprehensive documentation.

Agile Manifesto

Continuous attention to technical excellence and good design enhances agility.

Agile Manifesto

The natural output from high level design activity is some form of specification. This specification will:

• Share the solution across the team,
• Serve the solution up for critical review,
• Identify how the solution can be verified and …
• Define a plan for integration.

Often, recording the results of a face to face discussion takes a lot less time than writing the code and testing it over and again before we discover the right solution.

Given that the high level design identifies our components and interfaces, we can see how they contribute to user functionality, and how they depend upon each other. This visibility reveals a plan for realising the solution. From this enlightenment, we can load the backlog with tasks that allow us to build the solution incrementally, adding value to the code as we go. The design activity has therefore made it trivial for us to put well formed user stories into the backlog, and we’ve produced a high level specification along the way.

There are various tools that can help with this part of the process, such as the use case analysis (touched on in the first Agile myth) and various diagram notations. The design method itself deserves an article (or possibly a degree course) in its own right; but essentially we can’t start working on something unless we know enough about what we need to do, and how we’re going to do it.

Scrum mandates that every item in the backlog must be described to a sufficiently consistent standard that it can be delivered within a Sprint. This standard is the Definition of Ready, and it’s defined by the team to ensure that a task is worthy of being added to a Sprint. We know when we’ve done enough high level design, because we can probably write a user story (or technical task) with sufficient detail to meet our Definition of Ready.

Likewise, if we can’t meet our own Definition of Ready, then we’ve probably not done enough high level design. We may not have interpreted the user’s needs, or thought about the form and function of the solution enough for it to be clear. Design is a soft, thought-based exercise that touches no code and incurs relatively little expense. A project manager might refer to the outcome of our design activity as “achieving clarity” – when there are no more unknowns, and we’re ready to get on with it.

Just like building a wall, we can’t proceed if we don’t have the dimensions, the bricks or the mortar. In Scrum, a backlog refinement meeting should soon highlight whether a story is ready or not.

High level design identifies tasks to be added to the product backlog.

Agile Helpie

Plotting a route on a map determines how we will reach our destination before we start on the journey. Likewise, the high level design lets us work out what needs to be changed, and how, before we need to touch the code. Recording the components and interfaces we identify gives us a specification, and a plan to implement it. It’s a lot easier to write the right code first time when we have a viable plan.

It’s important to remember that we don’t have to complete the entire high level design to start writing code; but the activity itself should help us identify what we need to do first, and likewise what we can defer until later. The items we defer will likely depend upon the work we do early on.

Design the Solution

Just as the high level design lets us load the product backlog, detailed design activity is the method we use to work out what the right code actually is. We don’t want to draw a flow chart for every single method we right. We especially don’t want to do that after we’ve written the method. That would be silly, and that’s not what detailed design is for.

Detailed design activity allows us to define the solution for an individual task in the backlog. Once again, a defined solution allows us to formulate a plan for completing that task.

But wait a minute …

We favour responding to change over following a plan.

Agile Manifesto

I’ve just described how our design activity helps us define the items in the backlog, and helps us plot a route to delivering solutions for individual tasks. And yet, the Agile manifesto boldly states that we should prefer “responding to change”. Does this mean we should avoid all planning, spend our days writing code and wait for the issues to roll in? This item in the manifesto should be taken with a pinch of salt.

The manifesto is trying to make sure that we don’t try and plan our entire development schedule and stick to it rigidly. Instead, we want to deliver small increments, so that we’re nimble enough to handle new or changing requirements. The manifesto is not trying to polarise us into avoiding plans.

Agile practices encourage test-driven development, whereby we write the tests for our code before we write the code itself. That’s a great process – we know we’ve written the right code because it passes our tests. And by the time we’ve written it, we’ve already tested it. That’s even better.

So our process for writing the code right first time includes both coding and testing. But how do we know what tests to write? We don’t know what tests to write unless we’ve defined what we’re testing. To establish that, we need to do some detailed design.

From our detailed design, we should end up with some form of specification. This could be some diagrams, a document or even just a Gherkin script. As long as the information can be easily navigated and understood by the developers, and clearly specifies what the item(s) under test must do, we have a good specification.

I might stick my neck out at this point, and suggest that by adopting test-driven development, we are verging on that “comprehensive documentation” that apparently we do not favour. If a test documents the detailed behaviour of the solution according to the design, and we have comprehensive tests, then we probably have comprehensive documentation.

It’s not heresy to have comprehensive documentation, though. After all, if we have a well-defined process and we follow it, the quality of our product will be higher. And if the documentation drops out of the process like it should do, then we’ve not wasted a moment of our time.

Personally, I prefer some visual representation that allows me to understand the detailed design from the high level design that spawned it. That truly shares the understanding, and it means I don’t have to reverse-engineer thousands of lines of rotten code patches before working out what to do next. The more clearly we trace what we’re doing, the less likely we are to introduces errors or flaws during the highly technical and error-prone activity of writing code. The act of detailed design reduces the risk of bugs.

There are various ways to measure the standard of a design (again, this is an article in its own right), but SOLID principles and design patterns provide some good references. Crucially, considering the solution based on understanding gleaned from the high level design should take less time than writing and testing the code, and it should allow us to add subsequent features more easily. Therefore, we are more likely to write the right code first time.

So What’s Your Point …?

At the end of the day, we’re trying to create just enough of the right components that fit together well to fulfil the needs of the user – just like a Lego toy. The important thing is that we consider our actions before we act.

Methodical consideration of the design takes less time and effort than writing code. It makes plans more accurate and reduces the risk of bugs. What’s not to love?

Agile does not give us a process that enhances the quality of our product – it demands that we define a process that allows us to deliver products to the standard we require.

Whilst Agile encourages iteration, the intention is that each iteration adds value for the user. And whilst it encourages iteration, we want to minimise those iterations in order to save time and money.

The intention is not that we hack away indefinitely at the code until it vaguely does something we like. Often, developers believe that solutions are “discovered” in code over a number of iterations; or the solution “evolves” as if it grew out of something that was not under our control.

I’d like to think that the solution to a problem is something that should be under our control. We take control by doing just enough design.