Fifth Myth of Agile

Agile doesn’t work for embedded software development.

To an extent, like any myth, this can be true. But it’s not Agile techniques that don’t work. It’s a combination of other factors.

Thumbs Down

What can make Agile a touchy subject in the world of firmware development? What problems do we encounter when we’re developing a multidisciplinary product rather than a wholly software solution?

Bottoms Up

There’s an awful lot of embedded software (or “firmware”) out there – from toasters and door bells to vehicle braking systems and robots. Firmware exists at the “edge” of computing, where electronic circuits provide a connection between abstract data and the real world. Firmware developers need to know quite a lot about the circuits they’re driving, and how they interact with physical events.

The electronics’ designer needs to provide a platform of generic features for which there will be many applications; much in the same way that a hard drive stores files for many different purposes. Supporting such varied applications means that an electronic platform design is often deliberately “bottom up”. It’s up to the firmware to make the platform do something useful – whether that’s driving a solenoid to actuate a valve, or sampling an analog signal to receive data.

Electronic interface specifications are becoming increasingly complex. There’s enough complexity in a micro-processor manual to boggle the minds of the most experienced developers. The firmware engineer must ensure that thousands of individual bits within hundreds of registers are set correctly, at exactly the right time, in order to make the firmware affect (or be affected by) the outside world.

It takes a lot of focus to translate the behaviour of electronics into lines of code, and this can adversely affect the design process. Blinded by mnemonics and TLAs, code is written to drive the circuits – but it’s very easy to lose sight of what they’re being driven for. This lack of high level perspective can distort the design in such a way that higher level components (like public interfaces) become a secondary concern.

There is a darker impact on the code too. Bottom-up design is often backwards. We think about the technical details more than we think about solving the problems important to the user. We become conditioned to think about the code before the design. That’s backwards, and it’s not Agile.

Performance Art

Firmware is heavily constrained in terms of resources. It’s typically constrained to use kilobytes of RAM, consume micro-amps of current and run on megahertz of processor cycles. Embedded developers must therefore write high performance code, which makes it sound like an art form. Much like art, performance can easily become an obsession, just as much as it can be completely neglected. Both these outcomes can compromise the product for the user. The run-time performance is rarely something that a product owner will be able to describe in terms of memory, micro-amps or megahertz. Without this clarity, it’s all too easy to fall into a trap of paying far too much attention to performance; or nowhere near enough. That’s not Agile.

Even if we are given the luxury of precise performance requirements, it may not be easy (or wise) to deliver the required performance early on. Performance requirements that are specified to match a competitor without basis in technological capability can do more harm than good to product development. That’s not Agile either.

Reinventing Wheels

In the embedded world, we often seem to write code more than once. The same patterns occur, but the code is frequently written again for each product.

Often, it’s hard enough to write some code that configures the circuits to behave the right way at the right time, and then make this work with some application. Whilst we know that common code should be something we can pull out of a library and re-use, the firmware solutions often become tightly coupled to their hardware platform. We either re-write the same function for each new product, or copy a similar solution and tweak it until it works.

This often happens as a result of project-centric thinking – the mind is on delivering code on time and on budget, and copying a previous solution feels like it will take less effort to deliver. But in reality, copying a previous solution inherits all its debt – all the design decisions made under the pressure of another project, all the undiscovered bugs, and all those that we introduce by tweaking it.

We perceive that we do not have time to stop and think about creating re-usable solutions. But if we did stop, just for a moment, and think about these things, it would likely take us less time to integrate the same solution to any product, rather than spending excessive effort developing the same solution more than once.

Own Goals

Embedded systems typically depend upon proprietary electronic and mechanical parts. Whilst it’s possible to design these parts, their manufacture often depends on external factors, like component lead times and the order book of a manufacturer. It’s rare for organisations to make their own circuit boards and mechanical parts, so these things are not directly under the control of a project.

Such dependencies make it much harder to deliver a “potentially shippable product”. As I highlighted in the second Agile myth, it rarely helps anyone to release an embedded system in line with a Scrum sprint period.

Without hardware, we cannot integrate firmware. Early on in development, the goals of the electronic and software development teams are very different, and yet both must make progress towards the same goal. A typical issue in poor waterfall development is to deliver the electronic platform and then “paste” the software on afterwards. We may have designed the software earlier in the waterfall, but we can deliver nothing until the we receive the first prototype circuit board. This significantly elongates the delivery schedule, and integration issues are not identified until much later in the product development. That’s not Agile.

Thumbs Up

That’s enough about the bad stuff. Let’s look at some ways we can make firmware more Agile, as part of a wider product development process.

Top Down

As I pointed out in the third Agile myth, we often want to integrate our solution from the bottom up, in order to provide a firm foundation upon which we can add user functions more easily over time. But we need to approach the design of new products from the top down.

So what is top down design? How do we make sure we’re writing the right code at the right time?

The critical objective here is to establish what the user needs the product to do, and so the use case is our best friend. Use case analysis helps us identify the most important, high level functions of a system, and therefore how users (or other systems) interact with it. It’s a formal thought exercise, which captures a specification and shares understanding across the team. That specification provides a point of reference that will persist beyond informal conversations and transient developers.

The most efficient and effective method of conveying to and within a development team is face-to-face conversation.

Agile Manifesto

This is true. But whilst a conversation transfers information effectively, it does not record it. It isn’t there when a new developer joins the team, and it’s not there once we’ve left the meeting room. A use case (and any other specification) lets us keep hold of the discussions that we have.

Record the results of your analysis in a specification.

Agile Helpie

Given a set of use cases, we can decompose them into functional blocks across the physical constraints of our system. This structural analysis provides a much clearer plan for the code that we need to write. Patterns start to emerge, which highlight shared functions and common interfaces and help us quantify the effort required in the code. The form complements the function – which is the definition of good design. Even better, we work out what the code needs to do without needing to write any. It’s far more efficient than letting code “evolve” into the right solution.

The structural analysis that distributes functional responsibilities across the system identifies the critical dependencies for each function. These dependencies will determine the order in which we can develop the solution, and therefore contribute to a plan that supports incremental development.

The use case analysis reveals testable functional requirements, from which we can create one or more epics or user stories in our product backlog.

Establishing the right functional requirements and translating them into a viable product backlog is not a trivial task, though. It’s not something that can be achieved impulsively under a perceived pressure of time, so project management needs to allocate sufficient time to forming the backlog. In a typical product development model, this activity takes place during a definition phase. Crucially, though, to be Agile, we don’t need to define the whole product before we start building the product. We can have as many definition phases as we need, and whenever we need them.

Quality of Service

The performance of a product is often important to the user. From their perspective, a poor product will be described qualitatively – it will be clunky, slow, unreliable or run the batteries down.

From a developer’s perspective, it’s hard to design a solution in terms of qualitative expectations. We can only prove that something is sufficiently responsive, fast, reliable or economical by measuring it quantitatively. Furthermore, at the beginning of product development, it’s likely that neither the user nor the developer know which aspects of the solution will make it responsive, fast, reliable or economical. Agile practices let us deliver functionality quickly and incrementally, but functionality alone is not enough to meet the stakeholders’ expectations. If we focus on functionality, then we may tick plenty of boxes in the feature list. But it’s extremely inefficient to go back and rework the solution to improve its performance retrospectively. Rushing a poorly performing product to market can have adverse consequences for the business.

Early on in the product development, we need to translate the users’ expectations into performance constraints. What makes the solution responsive? How fast is enough? How long should the battery last? We need to spend some effort in establishing a performance envelope that can be verified, and that can be improved over time. The performance envelope defines formal requirements that we can prove.

Once the desired performance is established, it’s often difficult to achieve it on the first attempt. It also might not be feasible to hit the objective within the constraints of a project (i.e. time and cost). As we develop the functionality incrementally, so too can we develop the performance.

Agile tells us to avoid spending excessive effort on code that doesn’t provide value to the user. If a performance factor provides benefit to the user, then Agile encourages us to approach it iteratively. We can improve the performance over time – it really depends how important the performance is compared to the user functions. For example, we may tolerate a power-hungry solution in the short term, as long as it is functionally stable. The iterative approach allows us to define achievable short term goals, as long as they work towards the longer term goals of a road map.

Integration Milestones

Agile advocates that we deliver small increments in value frequently. We hear a lot about delivering a “potentially shippable product little and often” – but it would be naive to take this literally in the context of an embedded system.

Agile approaches can help us in two key ways when developing a new product:

• The Proof of Concept.
• The Product Backlog.

When we’re developing a new product, there will be a lot of unknown factors. To be confident that we can deliver a high-integrity product for real, it often helps to develop a proof-of-concept (PoC) or demonstrator model. The PoC development can use evaluation boards and open source projects to separate the boiler-plate solution from the user value.

Whilst this approach is a very Agile and effective way to prove a concept, it won’t deliver a product of any integrity. That is, an evaluation board running open source code isn’t really a shippable product.

Often, product development will not have the luxury of a dedicated PoC phase. A business may not be able to justify spending time and money learning how to make a product when it seems more efficient to just get started making the real thing.

Just as we can use Agile techniques to take us through a proof of concept phase, we can define goals that allow us to build the new product as we go. Similarly, by setting shared goals across all the engineering disciplines, the whole development team can work towards the same objective.

These goals allow us to remove unknown factors methodically. By defining strategic goals across the development program, we can …

• Gain clarity for the development program, which avoids downstream delays and pre-empts significant refactoring.
• Actively manage the inherent dependencies between software, hardware and sub-contractors.

We can get working on the easy stuff early on, and be confident that unknown factors will be resolved by the time they become important. In particular, Scrum encourages us to set specific objectives (or “spikes“) to learn more about the best solution to a problem. These can be added to our plans as risk factors, but they can be addressed dynamically if we know which aspects of the product they affect.

A product backlog is therefore loaded with known activities that, when completed in the right order, will remove unknown factors and deliver software and hardware components just in time for integration. The backlog contains tasks that guide us towards shared integration milestones.

Like the requirements analysis discussed earlier, loading the backlog is not a trivial exercise. Despite the Scrum approach of collective planning, it’s quite a skill to be able to load a backlog, and it’s unlikely this can be achieved within the short scope of a sprint.

Likewise. it’s very difficult for software developers and electronic engineers to share the same collective planning activity. So, in the embedded system, it helps to organise collective planning exercises strategically. Applying filters to a shared backlog and coordinating separate planning sessions can help here. In fact, this is exactly what frameworks like SAFe define – giving a bit of meat to the bones of the manifesto.

Creating a backlog demands significant up-front effort, following our initial structural and behavioural analyses. However, we don’t have to complete all the analysis to start working productively. Analysis that depends on the outcome of other tasks can be deferred until enough information is available. By that token, it’s important that we avoid loading the backlog with tasks that are beyond the horizon (i.e. further ahead than we can see).

Once an initial backlog has been established, it requires continual maintenance throughout the development cycle. A churning backlog makes progress visible. The nature of tasks shifts over time – from analysis, through realisation to integration. As we gain clarity, we also become more predictable. The negative view that projects that “always shift to the right” or plans that “don’t survive first contact” is therefore reversed as the backlog grows and shrinks to reflect the reality and pragmatism of development.

So What’s Your Point …?

The point is that Agile techniques can be employed effectively when we develop embedded systems.

Many of the techniques described here are just part of good project management, Agile or not. Agility comes from the whole team getting engaged with the system architecture, use cases, risks, dependencies and planning. The transparency of the plan shows progress to the stakeholder. Agile can improve how the whole product development team works together.

• Engagement: Break down silos, involve all disciplines in critical analysis and planning.
• Top Down Design: Define the use cases and distribute the functions across a system architecture.
• Risks: What are the unknown factors that will affect progress? Establish tasks in the backlog that remove the unknown factors.
• Milestones: Identify dependencies, establish shared goals, plan releases that allow support incremental development of features and performance.
• Backlog Maintenance: Filter the backlog for tasks specific to a given discipline (e.g. firmware, electronics), continually review the tasks and their sequence.