Posts Tagged ‘software development’

Turning the pyramid upside down

Thursday, November 6, 2014
posted by daveb

pyramidMuch has been written about managing change in large organisations. However, examples of successful change programmes are still far too rare. Even when change programmes succeed they take far too long, and cost far too much. Large organisations simply find it very hard to change the way they do things internally. This is because real change is impossible without the full buy-in of least a majority of the staff members affected. People simply don’t like change – there’s often much that’s threatening about the kind of change that gets pushed from the top down, and little upside. Put yourself in the position of an administrator that has become expert in a system over 10 years of daily usage. Now try telling that person that the system will be thrown out in favour of a new system. That is of course a very threatening situation to someone whose feeling of worth at the organisation is based on their indispensable knowledge of the old system.

Leading change in hearts and minds is surely the foundation of successful organisational change, because at the end of the day changing an organisation’s operations involves changing the way people do things. The key is leadership, in its classical sense. Unfortunately its just this kind of leadership that large businesses are frequently very bad at. How many examples of great leaders can you identify around you in your workplace? How many individuals can you identify that you would follow if your life depended on their leadership? A true leader has the respect of their team, and gives respect back. They don’t need to force their will on their team because their team trusts them and feels their point of view is heard. You should consider yourself fortunate if you can identify just one such person in your work environment, and sadly you would be extremely fortunate to work in a team with such a person in the lead role. It shouldn’t be like this of course, but that’s the reality of corporate culture.

But in stark contrast to this, “IT leadership” in large organisations typically decides on an IT strategy and starts pushing changes downstream onto the organisation. Business users typically get involved in the requirements gathering process, and “IT leadership” calls this “listening to the users”. But this is like me asking you: “What colour would you like your new car?” before I’ve understood your transport needs. Furthermore, I’m a car enthusiast, and I love the prospect of putting you in a new car (I know I’d want one, therefore you must too). Perhaps what you really needed was an annual train pass, but I jumped right over that because I have no interest in trains, let alone public transport.

We in IT leadership really need to engage with the business much earlier, and far more deeply – so deeply that we really understand what’s going on at the front lines of the organisation, where the rubber hits the road, as it were. But this is hard work. I’d argue that in any reasonably complex business environment it is not possible for analysts to understand everything that is going on at the business level, unless less they have been at the organisation for at least 3-4 years. The best analysts have been in multiple roles, including operational roles, so that have have a deep understanding of a business area in terms of operations, support, technology, and business process. These analysts are very rare. If you have analysts like these look after them! They are irreplaceable. Most analysts develop shallow approximations to what’s going on, and while these approximations can be useful, it’s very dangerous to view these approximations as if they are the complete picture. If IT leadership develops solutions based on these shallow approximations, they won’t build the right solution, at least not on the first attempt.

In the 1990’s IT got away with top-down driven change. E.g. we decided the company needed a PC on every desk, so we made it happen. But in 2014, things are no where near so simple. Our biggest IT projects aren’t often about providing this kind of basic infrastructure anymore – we already have that. What we need are new ways to do business, new ways to solve old problems, ways to get head and shoulders above the competition, ways to enable us to move faster. IT persists in thinking, like it did in the 1990’s, that it just needs to ask the business what the issues are, then come up with a snazzy technical solution.

However, there’s another way to approach organisation change, an approach that fosters trust and gets real buy-in, an approach that is borne of the Scrum, Agile and Lean movements. It is to turn around the notion that we in IT or organisational leadership actually even understand what changes would have the most benefit (even after asking the business), or whether change is required at all.

Instead we should eat some serious humble pie and try this: Ask the folks in the front lines themselves what needs to change, and critically, let them become the designers and leaders of change. Lead from behind by supporting them to get the changes they want by providing whatever they need – software, developers, analysts, trainers etc. but always work under the front line leaders (in Scrum, you’d make them the Product Owner). Turn the organisation chart upside down and respect and empower the front lines for that is where your organisation creates value.

Critical to this approach is the concept of iterations – the buildmeasurelearn feedback loop. This means we change something (preferably something small), measure its effect, and learn from that what to try next. Repeating this loop is the key. It actually doesn’t even matter what change you make – the measurement step will tell you whether it was a positive or negative change, and that will drive your next iteration. The iterations need to be as short as possible for this to be effective. One week iterations work well.

Of course it takes time for front line workers to get into this way of thinking (and longer for management!) after accepting decades of top down driven changes that have left them disillusioned and change-weary. But once they get the idea that they can have whatever changes they want, they’ll start flexing their new found powers creatively and working out how to make best use of it. There is nothing as empowering and engaging as being in control of one’s own success. And with that new power should come new responsibility – the front lines, not their managers should be responsible for their department’s results, because, after all, they are the ones creating those results, not their manager nor their manager’s manager.

The main drawback of the front-line driven approach I am arguing for is that big changes don’t happen that way. Finance systems don’t get replaced that way. Front-line staff are great at coming up with small improvements, but they may not be in the best position to see opportunities for big step changes.

The big changes may still need to come from IT leadership. But importantly, if IT staff are constantly working with front-line staff to make the small changes they need, then IT gains an enormous amount of understanding of the business during this process, and a huge amount of mutual trust is created if this is executed well. After 12 months of close work with the business users, IT staff are in a much better position to recommend larger “step” changes in the solution space.

Further reading about these concepts:

  1. Insecure Manager’s Don’t Want Your Suggestions
  2. Implementing Lean Software Development
  3. Peopleware: Productive Projects and Teams
  4. Extreme Programming Explained

Enterprise IT is broken (part 1)

Monday, November 18, 2013
posted by daveb

Broken windows in a St Petersburg abandoned cinema

 

You may have heard of Conways law:

“.. organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”

Conway’s law has proven to be true for every software project I have ever been involved in. Take the client-server application where the client was developed in C++ and the server in C. The client side developers were young and hip, and into OO. The server side developers were ex-mainframe developers in their 50’s. It’s fair to say the two parties did not see eye to about much when it came to software design. The friction between the parties came to a head in the shared communication libraries for client-server communication, which they co-developed. The libraries were littered with pre-processor definitions and macros that seemed to be at war with one another. The arguments between teams carried over into the version control comments. The shared communication libraries were some of the most un-maintainable, un-readable and bug ridden code imaginable, even though the purely client-side and purely server-side code were reasonably tidy on their own.

It was around 2008 when I began my adventures into this thing we call “the enterprise”. I was going to be an “enterprise developer” and take on development of a key part of the infrastructure – a credit card transaction processing interface. I understood that enterprise development meant you had to be lean – unlike a software company selling software products or software as a service, there were no economies of scale – you only build one instance of enterprise software.

As I began to find my way round some of the custom developed services and applications, a few questions started to emerge – like what version control system do you use here? Answer: “yes we have been meaning to get to that”. Ok, so there had been only one developer on that part of the project previously, and he was a bit green, so I decided I shouldn’t pre-judge the organization based on that alone.

More questions started to appear as my predecessor walked me through the operations side of things. He showed me how they had left an older version of the credit card processing API in production because there were an number of external clients using that interface, and they could not force them to upgrade to the new interface. Fair enough. I asked about where the source code was for the old version, in case I need to go back to it should a bug need to be fixed. Answer: “… well there shouldn’t really be any bugs, because it’s been there for years now”.

It turned out that work had started on “version 2″ without any kind of version control branch or label or even so much as a simple zip backup of “version 1″ source code. They had lost the intellectual property investment of the stable “version 1″, and had re-written large chunks of it to create “version 2″, which was not backward compatible, and was considerably less stable than the previous version. Unbelievable.

“Version 2″ had been 18 months in development, and had only very recently been deployed to production. Therefore, no business value had been delivered for 18 months. Business stakeholders had lost patience, and almost lost complete confidence in the development team.

Since the recent “version 2″ update, the phone had been ringing hot, and my predecessor would have an often lengthy discussion with an upset customer who had lost sales due to downtime and bugs with the service. I was now supposed to take these calls, and be an apologist for this mess.

At this point, things were looking so bad I was seriously considering walking out the door before I was even two weeks into the job.

However, I resolved to take on the project as a challenge, and that is the only thing that kept me there. I enquired about the testing approach: unit testing, integration testing, user acceptance testing and so on. In short:

Unit Testing: “what’s that exactly?”

Integration Testing: a spreadsheet containing a single sheet with dozens columns and a hundred or so rows, each representing a test scenario. It was un-printable, un-readable, inaccurate and was the un-loved product of something the boss had instructed the developers to do. The developers didn’t feel their job was to test the product, and instead of resolving this dispute with the boss, they had yielded to his pressure, but then done the worst possible job on it, to make the point that developers can’t test! This communication breakdown, and many other examples like it had almost eroded all value from the services being developed.

User Acceptance Testing: none

As we delved into architecture there were more surprises waiting for me. Like the messaging interface that used a database table as a queue, and had a service polling the table for new records every 500ms. This, I later discovered, would occasionally deadlock itself with another transaction and become the deadlock victim, meaning someone’s credit card transaction failed. The reason for using a table as a queue: the solution architect was a relational database guy and insisted this solution be adopted when the developers had hit some problems with the message queuing technology they were using.

Turns out there were more surprises in store

What is unbelievable is not that this dysfunctional situation could exist, but that project management and project stakeholders had no idea that these problems existed in the development practices and system architecture. They knew something was wrong, but lacked the ability to see any deeper inside the box. Nor did they have any notion that the industry has long since found best practices that address the very problems that were slowly destroying all value in the services they were providing.

At first I thought this organization was an anomaly, and that I would be unlikely to see anything this bad again. But then I started hearing about others who had seen similar things. And then I saw inside other organizations. I started to realize that what I’d seen was not an anomaly at all, it was practically commonplace. Sure, some were better than others, but I had yet to see inside an enterprise that had anything even remotely approaching a quality development team, with a solid set of practices that was able to deliver business value.

Conway’s law seemed to be holding true. Frictions between personalities and departments led directly to confusing and inconsistent software architectures. In fact Conway’s law can even be used in the reverse – where there exist strange inconstancies in a software architecture, you get an insight into the friction between different personalities or departments involved in its design.

If you want to assess your development team, and you aren’t a developer, just use the Joel Test. It goes like this:

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

Add one point for every “yes” answer and there’s your score. As Joel himself says:

” The truth is that most software organizations are running with a score of 2 or 3, and they need serious help, because companies like Microsoft run at 12 full-time.”