Agile should not be thought of as another project management practice, like an alternative to Prince II. Its a fundamental rethink of 20th century management thinking (mass production / economies of scale, efficiency, scientific management (Taylorism), planning, control, hierarchy and fixed roles) for the 21st century, where application of creativity + knowledge and empiricism (feedback from actual experimentation) are coming forward as the keys to success in a complex and chaotic environment of constant change.
The very idea that there needs to be a fixed scope “Project” in order to create value is under threat. If we can produce value in excess of costs (including opportunity costs), then why would we stop at some predetermined future point (project delivery date)? You’d surely keep going, unless of course you are not producing value in excess of costs, in which case why would you start?
In this sense Agile (+Lean) is about building the “manufacturing” lines of the future where it is not physical products that we are producing, but this ephemeral thing called customer value. A manufacturing line is nothing like a “Project” – it produces value as long as customers are buying, and this is what Agile/Lean is really about.
As an industry, we have tried just about every method imaginable to deliver software. For every methodology that claims higher success rates it’s easy to find counter examples where the same methodology has been a failure.
I think what The Manifesto has done well is to point out that the things successful projects have in common are not in their method, but a set of very people-centric values and principles. Basically the underlying message in The Manifesto is that people matter more than process.
We need to stop talking about people as “resources”, and see them as individuals with character, intelligence, and a desire to make a contribution. We need to collaborate, communicate, bring customers and developers together, trust one another, allow experimentation, mistakes and therefore learning to take place. Allow autonomy, mastery and purpose and great things can happen.
Kanban is an ideal candidate to use with a “Release Train”. The Release Train works like a freight train that leaves every week at the same time. The user stories that are tested and ready to ship go on the train and get delivered. If a story is not quite ready we don’t work overtime nor do we change the ‘train’ schedule, we just ship whatever is ready to ship at that time, and if something misses the train there’s another one next week. Occasionally a ‘special’ train runs between the normal scheduled times because there is something of value to deliver.
Some stakeholders/customers may want to know exactly when a particular feature will ship and the down side of this approach is we don’t really make such promises. What we do though, is to build trust – stakeholders don’t tend to hover about asking what the team is working on when quality software is shipping on a regular basis.
This approach is based on the idea of Optimizing the Flow of Value to the business/customer. If we deliver to the customer feature B instead of feature A this week, they still get to see a stream of value flowing their way, and that is something that makes customers happy.
If there is a measurable definition of what Agile is, Cycle Time is it. Cycle Time is Agile distilled down to it’s very essence, with all the sentiment stripped away. To be Agile is to be responsive. Responsiveness means keeping up with the business, because if you are not keeping up, you are lagging further and further behind.
If I could choose just one measurement of Agile success (and it’s probably a good idea to have just one measure) I’d choose Cycle Time – that is time (in days) to turn a request or requirement into delivered business value (i.e. in production).
This is a very objective measure that is hard to kid yourself on. It’s also easy to measure, and it has direct meaning to stakeholders, business users/customers and team members alike.
So to test your success in becoming more Agile (whatever that means) test your Cycle Time before and after your transformation. This is easy enough to do – take a look at how long your average feature request took from inception to production before and after your transformation.
Dozens of great books have been written around product and software development with lean/agile and related concepts over the last 50 years or so. While Agile is still young and evolving rapidly (bear in mind the term Agile was only coined in 2001) Lean and related concepts like the Theory of Constraints pre-date that term by decades. The Agile Manifesto was not the invention of iterative, business-focused development processes, it was just a milestone in its evolution towards mainstream adoption in software development. Very seldom have I seen an online resource even mention this fact – I guess the Internet believes history started in around 2000!
A small, random scattering of books to get some history on Agile and some insight into the origins of Lean and Theory of Constraints:
The Toyota Way
The Goal (Goldratt)
The Mythical Man Month (Brooks)
Extreme Programming Explained (Beck)
Lean Software Development (Poppendieck)
The Phoenix Project
Toyota Kata (Rother)
The Lean Startup
Hopefully some of these will lead you to deeper insight than online resources and get you past the hyperbole of Agile, towards seeing the 50+ years of solid evidence that supports its underlying hypothesis.
Agile, as in The Agile Manifesto (http://agilemanifesto.org/) is a very elegant way to express the learnings of many software experts over many years. The Agile Manifesto is a kind of declaration by leading software experts that there is another way to think about software development other than classic waterfall based, documentation / process heavy methodologies that tend to see people only as resources. While these experts don’t attempt to put forward a single recommended methodology, they do agree on the principles of the Manifesto, and indeed they coined the term “Agile”.
The Agile Manifesto hints at what Agile is by comparing the Agile way to the traditional way – “Individuals and interactions over processes and tools” for example. If you haven’t heard of the Agile Manifesto, chances are you’ve picked up some highly distorted definition of Agile, such as “Agile means no documentation” or “Agile means Scrum” or “Agile means iterations”. The Agile Manifesto says nothing of the sort.
So to respond to the question “What’s the biggest weakness with Agile?” we have to first be clear about what we mean by Agile. Ideally, we’d use the definition from The Agile Manifesto, but unfortunately the term Agile has become very overloaded, and has been mis-applied and widely misunderstood. The real weakness with “Agile” (in the sense its commonly used today) is the very word itself! Its almost unusable due to its many interpretations, connotations and so on.
For example, if you are trying to convince your organisation that it could benefit from Agile, its probably safer to avoid the term altogether than to risk those that have a definition of Agile in mind misunderstanding you completely.
Allow me to share a question, and my response to it on Linked-In’s Agile Coaching group.
Do you have any ideas or suggestions on how to improve the team focus while the Scrum (Ops) team works on several changes that sometimes don’t have a relationship?
Context: A mainframe environment of >6500 COBOL programmes where several highly complex regulations are gathered in one system.
We have a very similar problem – one Product Backlog, one Scrum Team but the work that some team members do has little to no impact on what other team members do, to the point where they have independent test/release cycles. There’s probably 3-4 separate sub-teams, sometimes with overlapping resources, working on separate products that occasionally interact.
In Scrum terms, Scrum of Scrums is the advice on how to scale Scrum. This may be of benefit to you if you have say > 20 people involved, but for us with 9, it seems like too much ceremony.
I think there is perhaps an absence of good advice from Scrum around how to run Scrum with a small team & a diverse range of products. Scrum can still work, but there’s more waste in the form of exposure to communications that have no bearing on your work.
I would question whether Scrum is the right fit. Scrum has a sweet spot that is around complex (single) product development efforts with team(s) of 3-9 members, where much of the work is in-sourced, and the team is co-located.
It may also be worth looking at Kanban which has its sweet spot in product maintenance and support, because, chances are, if you have a small team and a diverse product portfolio to manage, you are in BAU mode.
What happens when Requirements Analysis, Design, Development and Testing all happen at once, on the same features?
Common sense software project management says that Requirements Analysis, Design, Development and Testing are distinct Software Development Life Cycle (SDLC) phases that happen one at a time, each being dependent on the step before it. This is what we commonly refer to nowadays as Waterfall (before Agile I believe we simply called this Project Management).
Agile & Lean based methodologies point out that we can view a software product as a group of features, order them according to business value, then start work on Requirements Analysis for the most high value items first. We then pass those over to Design & Development while we are do Requirements Analysis for the next most high value feature, and so on, creating a pipeline with different features at different stages of delivery (a value stream).
But that’s not exactly what I’m talking about here – I’m talking about doing Requirements, Design, Development and Testing for a single feature at the same time.
Common sense also says that doing all of these things at once is not only a bad idea, but simply not possible. For instance, how can Testers test software that has not been Developed yet? (This is a common assumption that is incorrect, because testers have a lot of value to add to a team than simply testing).
Therefore, I was surprised to find that on a project I am currently working on the team had self-organised to do all four activities at the same time, for the same feature. And you know what? It was the most productive we have been in months!
So how did this work? Admittedly, Requirements Analysis had a head start on the other activities. Design commenced with the Requirements in an incomplete state – they were more or less correct, and covered most of the ground necessary to describe the the business need, but completely lacked the depth of analysis necessary for the Design, Development & Testing activities. The Designer had no choice but to start by clarifying the Requirements i.e. the Designer was asking a lot of pointy questions of the business to make sure they had thought through what they appeared to be asking for, and looking to flesh out some shallow requirements with greater depth.
Meanwhile, the Developers had been able to make a start on the infrastructure parts of the solution, and they were able to start building and refining the solution as new information and clarity came from the Designer’s work. The Design & Development team, and in fact all team members worked very closely together. Most communication was verbal.
At the same time the Tester was able to start working with the Designer and the Business Analyst in charge of the Requirements to start gaining an understanding of how the system was supposed to work, and what he would need in order to test it. He was able to start doing productive work well before the Development Team were anywhere near completion, and was able to question the Requirements and add clarity there.
The result: so far this appears to be working extremely well, although I won’t declare it a success until the project is delivered. The key improvement over the staggered waterfall approach is the quality and intensity of communication within the team. Everyone is talking about the requirements, the design, and how to test it, all the time. It’s a real buzz to work in an environment of rapid-fire discussion, thought and communication and it helps bring out the best everyone has to offer. This is how a team would work if it had an impossibly short time frame to deliver, and their lives depended on getting it done!
The irony of this is that we are supposed to be working in a Waterfall project management environment – but that was working so badly that the team members basically self-organised and did whatever gave us the best chance of success with a deadline rapidly approaching, and all eyes on our team.
In order to make this “Do everything at once” approach work, you need the following:
- Co-located, small (3-9 member), cross-functional team.
- Safe communication environment – anyone can say what they need to when they need to. There is respect for all opinions, no danger of a boss overhearing and suggesting we are not doing things the “right way”.
- Professional, transparent, capable team members in every discipline.
- You need a coordinator role on the team. I suggest the Designer fulfills this role since they are in the best position to bridge all the disciplines. This role needs to communicate the constant stream of changes to all the team members.
- No formal change control processes around the Requirements & Design activities.
- No blame. Instead of blaming the business for its poor requirements or the lack of design documentation, dive in and make them better.
Would I recommend this approach? No, not as general advice, but in the right circumstances, it is definitely worth a try. At the very least it’s worth a try if you are falling behind schedule, everyone is blaming everyone else, no one is talking etc. – what have you got to lose?
While Scrum can claim great success in its adoption rate on product development and software projects, I would put forward that many of these Scrum projects, particularly the software projects, have not been anywhere near as successful as anyone had hoped. I put forward, based on my own observations, that many attempts to install Scrum (especially on enterprise software development projects) have resulted in a gradual reversion back to pre-Scrum practices. These Scrum teams, if they are honest with themselves, may declare Scrum only a partial success at best. (If they are not, they will declare Scrum a success anyway!).
Agile practices, on the other hand, as originally outlined in the Agile Manifesto, and before that in Kent Beck’s work on Extreme Programming (refer Beck’s book eXtreme Programming Explained) have not been anywhere near as successful in their reach into mainstream enterprise culture as has the practice of Scrum.
Frequently we see Scrum adopted on software projects without co-adoption of Agile practices (continuous integration, automated testing, refactoring, etc.) This is unfortunate because an iterative project management practice (Scrum) imposed upon a software project, without the adoption of suitable software practices is a recipe for disaster.
Scrum teams that have not developed the Agile practices (and technical skills) that enable their software to be continuously modified will quickly find out that their code base collapses under its own weight, as iterative change piled on top of change decays the quality of the code, and gradually makes future change more and more difficult. Martin Fowler warned us about this is his blog post titled Flaccid Scrum.
However, the Scrum “brand” continues to be strong, perhaps because of the success of Scrum in product development, and areas outside of software development where the lack of Agile technical practices is less relevant. The Agile “brand” (perhaps because of its direct connection with software development) on the other hand has taken the hit, with Agile almost a dirty word in some corporate environments.
This is the true tragedy. I put forward that it is much better to have the Agile practices adopted by your software team than it is to have achieved a Scrum implementation devoid of the necessary Agile technical practices. Scrum without Agile practices is a Pirrhic victory.
Much has been written about managing change in large organisations. However, examples of successful change programmes are still far too rare. Even when change programmes succeed they take far too long, and cost far too much. Large organisations simply find it very hard to change the way they do things internally. This is because real change is impossible without the full buy-in of least a majority of the staff members affected. People simply don’t like change – there’s often much that’s threatening about the kind of change that gets pushed from the top down, and little upside. Put yourself in the position of an administrator that has become expert in a system over 10 years of daily usage. Now try telling that person that the system will be thrown out in favour of a new system. That is of course a very threatening situation to someone whose feeling of worth at the organisation is based on their indispensable knowledge of the old system.
Leading change in hearts and minds is surely the foundation of successful organisational change, because at the end of the day changing an organisation’s operations involves changing the way people do things. The key is leadership, in its classical sense. Unfortunately its just this kind of leadership that large businesses are frequently very bad at. How many examples of great leaders can you identify around you in your workplace? How many individuals can you identify that you would follow if your life depended on their leadership? A true leader has the respect of their team, and gives respect back. They don’t need to force their will on their team because their team trusts them and feels their point of view is heard. You should consider yourself fortunate if you can identify just one such person in your work environment, and sadly you would be extremely fortunate to work in a team with such a person in the lead role. It shouldn’t be like this of course, but that’s the reality of corporate culture.
But in stark contrast to this, “IT leadership” in large organisations typically decides on an IT strategy and starts pushing changes downstream onto the organisation. Business users typically get involved in the requirements gathering process, and “IT leadership” calls this “listening to the users”. But this is like me asking you: “What colour would you like your new car?” before I’ve understood your transport needs. Furthermore, I’m a car enthusiast, and I love the prospect of putting you in a new car (I know I’d want one, therefore you must too). Perhaps what you really needed was an annual train pass, but I jumped right over that because I have no interest in trains, let alone public transport.
We in IT leadership really need to engage with the business much earlier, and far more deeply – so deeply that we really understand what’s going on at the front lines of the organisation, where the rubber hits the road, as it were. But this is hard work. I’d argue that in any reasonably complex business environment it is not possible for analysts to understand everything that is going on at the business level, unless less they have been at the organisation for at least 3-4 years. The best analysts have been in multiple roles, including operational roles, so that have have a deep understanding of a business area in terms of operations, support, technology, and business process. These analysts are very rare. If you have analysts like these look after them! They are irreplaceable. Most analysts develop shallow approximations to what’s going on, and while these approximations can be useful, it’s very dangerous to view these approximations as if they are the complete picture. If IT leadership develops solutions based on these shallow approximations, they won’t build the right solution, at least not on the first attempt.
In the 1990’s IT got away with top-down driven change. E.g. we decided the company needed a PC on every desk, so we made it happen. But in 2014, things are no where near so simple. Our biggest IT projects aren’t often about providing this kind of basic infrastructure anymore – we already have that. What we need are new ways to do business, new ways to solve old problems, ways to get head and shoulders above the competition, ways to enable us to move faster. IT persists in thinking, like it did in the 1990’s, that it just needs to ask the business what the issues are, then come up with a snazzy technical solution.
However, there’s another way to approach organisation change, an approach that fosters trust and gets real buy-in, an approach that is borne of the Scrum, Agile and Lean movements. It is to turn around the notion that we in IT or organisational leadership actually even understand what changes would have the most benefit (even after asking the business), or whether change is required at all.
Instead we should eat some serious humble pie and try this: Ask the folks in the front lines themselves what needs to change, and critically, let them become the designers and leaders of change. Lead from behind by supporting them to get the changes they want by providing whatever they need – software, developers, analysts, trainers etc. but always work under the front line leaders (in Scrum, you’d make them the Product Owner). Turn the organisation chart upside down and respect and empower the front lines for that is where your organisation creates value.
Critical to this approach is the concept of iterations – the build – measure – learn feedback loop. This means we change something (preferably something small), measure its effect, and learn from that what to try next. Repeating this loop is the key. It actually doesn’t even matter what change you make – the measurement step will tell you whether it was a positive or negative change, and that will drive your next iteration. The iterations need to be as short as possible for this to be effective. One week iterations work well.
Of course it takes time for front line workers to get into this way of thinking (and longer for management!) after accepting decades of top down driven changes that have left them disillusioned and change-weary. But once they get the idea that they can have whatever changes they want, they’ll start flexing their new found powers creatively and working out how to make best use of it. There is nothing as empowering and engaging as being in control of one’s own success. And with that new power should come new responsibility – the front lines, not their managers should be responsible for their department’s results, because, after all, they are the ones creating those results, not their manager nor their manager’s manager.
The main drawback of the front-line driven approach I am arguing for is that big changes don’t happen that way. Finance systems don’t get replaced that way. Front-line staff are great at coming up with small improvements, but they may not be in the best position to see opportunities for big step changes.
The big changes may still need to come from IT leadership. But importantly, if IT staff are constantly working with front-line staff to make the small changes they need, then IT gains an enormous amount of understanding of the business during this process, and a huge amount of mutual trust is created if this is executed well. After 12 months of close work with the business users, IT staff are in a much better position to recommend larger “step” changes in the solution space.
Further reading about these concepts: