The new design methodologies

This article first appeared in issue 224 of .net magazine – the world's best-selling magazine for web designers and developers.

Until recent years, the traditionally accepted process for software and web development has been the Waterfall methodology. Increasingly however, Agile software development methodologies have gained traction – including, among others, Scrum, Extreme programming (XP), Lean software development and Feature-driven development.

But what exactly are they? Where did they come from? And could a digital agency have any use for them to better manage projects?

To answer that, we need to take a step back and look at today’s rapidly shifting internet culture. We live in a world with loads of user-generated content and self-publication, a rise in the adoption and acceptance of social networks and an increasing app-based economy. This means that, as digital agencies, we need to sit up and take notice of changing online behaviour more than ever before and use it to inform the projects we work on.

The fact is that, as a digital agency, we are increasingly being asked questions along the lines of: “do you ‘do’ Agile projects?” And in particular, “do you know Lean?”

Learning from the past

This might come as a surprise, but these methodologies are not new. Agile can be traced back to the 1970s and Lean goes back even further to the 1940s, when a Japanese car manufacturer developed it from supermarket processes as a way of improving production line performance. Today, these two models are being increasingly adopted by digital agencies. But why?

The traditional Waterfall methodology is being put on the shelf. In simple terms, this process sees a project divided into distinct phases of requirements; design, build, test and release. Don’t get me wrong; there are benefits in the simplicity of this approach, to ensure your project delivers against its initial requirements. But the issue is that, with Waterfall, what you deliver at the end of the project may no longer meet your customers’ real world requirements, despite what was agreed upfront.

This is where Agile and Lean really come into their own. And, if you combine the two, these methodologies can help deliver remarkable results.

Limbering up

Agile builds the idea of ‘change’ into the process itself. It allows for flexibility to help brands respond to customer feedback, changing business priorities and innovations driven by the internet. Technology can be released incrementally to get feedback from all parties: customer, client, and agency. Agile’s guiding principles can be summarised as follows:

• Iterative releases (Sprints) of useful and meaningful work to your client.
• Requirements are prioritised and low priority items can be descoped if there are increases in cost or to hit deadlines.
• Adaptable to changing requirements, even at a late stage.
• The measure of success is working software delivered to the client.
• Progress can be measured using Sprint Burndown charts, which continually track the amount of outstanding effort across all remaining work items.
• Daily cooperation between developers and client.
• A focus on the capability of people, not a restrictive process.

Removing the fat

But, on its own, we found that pure Agile (Scrum) simply isn’t agile enough for our agency! This is where Lean comes in. It works on the basic principle that once an item of work is built and tested it can be put live, either individually or as part of a batched release.

This methodology adheres very closely in concept to Lean manufacturing principles. It eliminates waste, amplifies learning, helps make decisions as late as possible, delivers as fast as possible, builds integrity into the process, empowers a team and enables a holistic overview of progress. Just to add a little confusion, Lean is often considered as a subset of the Agile methodologies, but to help you understand better, here are the guiding principles:

• Requirements are broken up in to work items (aka User Stories).
• Work items are placed on the Lean Kanban board, which plans and tracks its journey through your internal project lifecycle on a daily basis.
• Sprint Burndown charts can be used to track progress; however the Kanban board is often sufficient.
• Each work item is designed, built, tested and shipped individually to deliver true flexibility.
• Daily cooperation exists between developers and customers.
• Customers are empowered to decide on work item priorities on an almost daily basis.

The ability to remain flexible and build changes into a development process makes Agile and Lean powerful tools in this fast-paced digital world. In the next sections we’ll explore more of the pros, cons and practical examples of application of Agile and Lean that are, in my opinion, here to stay.

Pros and cons of Agile, Waterfall and Lean

But before running off and pushing Agile and Lean onto every project you work on, you should take some time to consider the merits of each methodology. In fact, choosing the right methodology for your project or your company is definitely one of the biggest challenges in ensuring you fulfil the brief.

In order to do this, you need to combine a good understanding of the pros and cons of each option with an honest assessment of your own internal processes. It’s important to assess why you should move from one methodology and, if you do, which elements of the previous methodology will serve your new approach and which ones you might want to carry over.

To help with this, I’ve outlined a high level matrix listing pros and cons for each methodology:

Pros

Waterfall

• Easy to understand and most developers have experience with the process.
• Requirements are nailed down from the start.
• Work is designed, built and tested in distinct project phases that can easily be tracked.

Agile (Scrum)

• Daily collaboration with developers and the customer enable the team to self-manage.
• Flexibility increases through iterative releases of meaningful work shown regularly to the client.
• Risk decreases as work is signed off at intervals.

Lean

• Total flexibility and able to deliver individual work items as and when they are ready.
• Requirements of individual items can potentially change right up until they are built.
• Lean project process can be mapped directly to your internal company processes.

Cons

Waterfall

• Risk where once requirements are defined, client doesn’t see work until User Acceptance Testing.
• Lacks flexibility to react to changing requirements.
• Work items that are complete have to wait until the next project phase before they progress.

Agile (Scrum)

• Incorporates a large number of terms, which can alienate project stakeholders (including Scrums, Sprints, Burndown charts, User Stories, Show and Tells).
• Work items need to be closely managed through iterations to prevent overloading the last release.
• The need to resource bug fixing at the same time as development.

Lean

• With improved flexibility comes greater responsibility to closely manage work items and overall project progress.
• High level project plans, a Kanban (workflow) board and Burndown charts can be used as tools to better manage the project.
• A key consideration is to resource bug fixing in parallel to development.

Case studies: the difference between corporates and agencies

Once you have an understanding of which methodology you want to adopt, and the key elements of each you want to use, the next step is understanding how to apply it practically.

I’m lucky that I have a broad background in software and web development, having worked for many years in small software houses, large corporates and digital agencies. In my previous role as a development manager in a large corporate financial institution, I led the introduction of Agile into our online banking development division on a multimillion-pound regulatory project.

I found that, out of the box, Agile processes work well in a corporate environment. A key point to note was that we still used the Waterfall approach to capture requirements and produce terms of reference to explain how we would deliver the project. Thereafter, through design, build and test we followed the standard Agile lifecycle, where all three phases run in parallel with iterative releases to the customer every two to three weeks throughout the project.

The customer feedback was extremely positive: they enjoyed the Show and Tell demonstrations that accompanied every release, because they made them feel as if they were influencing our work as it was being built. They liked the fact that, rather than IT disappearing into a black hole for eight months, they could see our changes within four weeks of the project initiation.

The most valuable lessons I learned through this experience could be summarised as:

• Get to know the process inside out and detail the pros and cons. Make sure that you’re doing it for the right reasons and that your team has totally bought into it.
• Especially in a corporate environment, be prepared to spend significant amounts of time presenting the process and selling its benefits to a wide range of stakeholders.
• When working alongside other development teams, ensure you clearly communicate your dependencies and map your project milestones across teams, particularly if they continue to use the Waterfall approach.

Things change

When I first joined a digital agency, my immediate thoughts were that Agile would be a good fit. As with any process, it’s about taking the things out of it that work for you. But, to be honest, I wasn’t convinced Agile (Scrum) could work because of the following factors:

• While more flexible than the Waterfall lifecycle, it ironically isn’t ‘agile’ enough for an agency.
• Agile promotes schedules of releases (Sprints) with fixed dates and cycles, but in agency work we offer more flexibility.
• In my experience, Agile is more suited to projects with over one month’s work. In an agency we deal with projects from an hour to a year, so we need to be more flexible.

After my first few months of agency project delivery, my fears were proved both right and wrong. It was clear that the expectation of an agency is to be as flexible as possible – and certainly beyond the level that Agile can achieve.

The key difference with a corporate company is that design agencies have to manage multiple projects, all with different timescales and complexities, which are delivered through the use of flexible resource pools.

The biggest challenge for the Agile methodology is the ability to deliver to a fixed release schedule when resources, projects and timescales are continually changing.

This is where Lean comes in to play. Lean offers complete flexibility, enabling the agency to deliver work items rapidly as and when required throughout a project.

When I first joined Realise Digital we were using large whiteboards for each team to manage their resourcing for the week. Interestingly these whiteboards weren’t a million miles away from the Lean Kanban board. This is a board (either physical or software-based) that enables you to break down a project into small work items and stick them on the board in a column that indicates progress in the overall project lifecycle. Even better is the ability to customise a Kanban board in order to meet exact internal processes (regardless of whether they are even documented).

In many ways, agencies and corporate companies can be considered opposites when it comes to project management.

Large corporate entities have to deliver projects to a megalithic scale, using consistent standards and processes. Agencies, meanwhile, need to deliver small to medium sized projects to a high standard while adapting to changing requirements and workload peaks.

Agile/Lean best practices

Based on our experience of working with Waterfall, Agile and Lean methodologies, here are a series of best practices that should prove useful for any digital agency looking to adopt different project management processes:

Communication – keep talking
Daily stand-ups (aka Scrum stand-ups) are a useful way to keep up to date with your project team and your client. For small project teams under five people, this can be done informally through the day. For teams larger than five, it’s worth getting together and spending a couple of minutes each providing an update and taking any urgent risks or issues offline for a follow up meeting. Keep the client involved/updated either daily, every second day – no longer than each week – otherwise they will wonder what’s happening!

Risks and Issues – but it might never happen
No project is perfect. There are always risks and issues. It’s best is keep on top of risks on a daily basis and manage issues urgently to get quick resolution to problems.

Keep a tracker and ensure your client understands it – it’s always good to be open and honest with your clients.

Requirements – get it right from the very start
It’s true that projects only go wrong from the start, so don’t wait until the end to find out. Make sure that you spend enough time detailing and prioritising requirements.

Planning – I love it when a plan comes together
Whether using Waterfall, Agile or Lean, a high-level project plan should be used to agree and track milestones with the client. Keep it up to date and to an appropriate level of detail.

Resourcing – you can’t beat continuity of resource and good communication
Resource pools across various teams provide excellent flexibility and maximise profit; however there’s no substitute for maintaining consistent team members throughout the project. This reduces the need for handovers and lengthy documentation while maintaining knowledge of the project and keeping consistent customer communication.

Design – make sure it’s adding value
Determine what documentation you require for your project to get your developers started. Continually review and update the documentation throughout the lifespan of the project and add more detail for the more complex or risky areas as they are developed. At the end of the project, don’t forget to update documentation to ensure that it reflects what was delivered.

Build – you know what you have to do
You can start development on the smaller, simpler work items immediately after you have agreed the requirements with the customer. This can be done in parallel with gathering the requirements for and documenting the design for more complex items.

Agile/Lean iterative releases – set and manage customer expectations to avoid surprises
Whether you release a small single work item or a batch as part of an iterative release, ensure you tell your client when you’re doing it, what is going to be delivered and what isn’t yet completed. You’d then be able to wrap this up in a release note and include any known bugs. This will avoid any surprises.

Testing – run in parallel with design and build
If you’re accustomed to the Waterfall approach, it’s likely to feel strange to be developing new functionality at the same time as testing out what you’ve already released.

The best way to manage this is to stay on top of your defects on a daily basis – for example by updating your Kanban board with your project team first thing each morning.

Make sure that you always leave reasonable time at the end of the project where you have stopped development and you focus purely on testing the final release.

Implementation – are you ready?
If you release small work items on an individual basis, it is still well worth putting together a release note including a high-level release plan. You’ll then be able to share this with the client in order to help manage expectations.

For larger releases, or the final project go live, put together a detailed release note and plan. Make sure that the client signs off the plan and is clear when they need to test your release once it’s gone live. Finally, don’t forget your back-out plan!

Post implementation – hope for the best, but prepare for the worst!
Be certain you’ve prepared sufficient cover on the following day to ensure that any warranty issues can be addressed in a timely fashion. Stay in touch with your customer and keep a close eye on your new release.

In the final analysis, it’s not a question of which methodology is “the best one”, but understanding how each model works and what it has to offer your specific company or project. Whether you go for Lean, Agile. Waterfall, or a combination of elements from each, it’s all about finding what process works best for you, and communicating it effectively to your teams. This may take a bit of thought and work, but in the end the benefits will be obvious.