Warning: include(wp-includes/class-wp-query-update.php) [function.include]: failed to open stream: No such file or directory in /home/content/06/8316506/html/wp-settings.php on line 376

Warning: include() [function.include]: Failed opening 'wp-includes/class-wp-query-update.php' for inclusion (include_path='.:/usr/local/php5/lib/php') in /home/content/06/8316506/html/wp-settings.php on line 376

Warning: include(wp-includes/class-wp-term-connect.php) [function.include]: failed to open stream: No such file or directory in /home/content/06/8316506/html/wp-config.php on line 92

Warning: include() [function.include]: Failed opening 'wp-includes/class-wp-term-connect.php' for inclusion (include_path='.:/usr/local/php5/lib/php') in /home/content/06/8316506/html/wp-config.php on line 92
Software Development « Ocean Web

Archive for the ‘Software Development’ Category

The Problem Space

Monday, June 30, 2014
posted by daveb


Guess which requirement was written by a solution-space thinker, and which was written by a problem-space thinker:

Requirement A: As the Iceberg Manager I need to receive timely (same day) notice when an iceberg breaks free of the ice shelf so that a decision can be made on whether or not to track it.

Requirement B: As the Iceberg Manager I need the system to provide an online form for the Iceberg Scientists to fill in when an iceberg breaks free of the ice shelf.

The trouble is, requirements like B seem to riddle project/product backlogs. It is solution-space thinking, and its poor work if done by a professional Business Analyst. (As an aside, if your Business Analyst or Product Owner writes requirements like B above, you need to tell them, or at least tell the Project Manager, that the whole team is being excluded from the creative process by such “requirements”.) Its leads us to a solution, short circuiting the potential to explore the problem from the many creative angles that a typical software development team is capable of. It fails to describe the nuances of the problem at all.

On a project team, this effectively shuts down all creative thinking, except that which comes from the Business Analyst. Software Developers, Testers, Technical Writers and other technical people are intelligent and creative people – that’s how that got where they are. It’s a terrible shame to exclude them from the creative process, and the quality of work, not to mention the morale, of the team will suffer greatly. Business Analysts and Product Owners are creative intelligent thinkers too, but we really need them to take on the role of defining the problem as clearly as possible. There’s no reason they can’t be part of the solution design, but their first job is to define the problem-space clearly. And when they move into the solution-space, they should expect their voice to be one of many, and that it should carry no particular authority over the team.

Hence we need the Business Analyst or Product Owner more than ever to think in the problem-space. If you don’t have a dedicated Business Analyst or Product Owner on your team, you have to be especially careful to avoid the traps of premature solution-space thinking, which frequently happens when a team of solutionists hear about a problem that needs solving. Being solutionists, we like our solution to be the one that gets implemented. This means we have to be quick to voice our solution as early as possible in order to get the buy-in of the other team members. This is wrong-headed in itself and is a sign that a team is a group of competitive, self-interested individuals rather a team at all. But worst of all, it drives the team to bypass the stage of really thinking about, and analyzing, the problem-space properly. And when we do that, we risk it all. We risk solving the wrong problem, or completely missing simple solutions, in favor of those that first come to mind, or those that are the easier to convince the team to agree to.

I’ve been involved in plenty of projects where I’ve worked directly with the customer / entrepreneur / stakeholders, without a Business Analyst in the middle, so this is nothing new. However, I have come to value the input a good Business Analyst brings to the project more and more over the years.

There’s a point made clearly in the Lean software development process that goes something like this: “The largest source of waste in software projects is building the wrong thing”.

Its kind of obvious when you think about it – should your team spend 3 months building a solution that does not solve the business need, and does not end up getting used, you’ve wasted 3 months effort. Its as if you achieved nothing at all in 3 months. You would have been better to send the whole project team away for a 3 month (paid) holiday. At least then they would have come back fresh and full of new energy. But I digress.

Getting back to the Business Analyst (or the equivalent person in the problem-space mindset), the value they add is having a mindset purely focused on the problem-space. We in technology are so deeply involved in the solution-space that we find it hard to work in the problem-space in an open minded way. When technologists, project manager’s, and others embedded in the solution-space attempt to write requirements that define the problem-space, we do a rotten job because we usually have a solution in mind as we are writing. We then reverse-engineer the requirements from the solution such that the reader will be led towards our solution. Exploring the problem-space fully is essential if we are to be lean, because being lean means building the right thing, and building the minimal thing that will solve the problem. Defined that way, we’d be well advised to be very critical of the definition of the problem in the first place. One poorly chosen word might lead us towards unnecessary complexity.

So where have we got to with all this? We solutionists should take all requirements with a grain of salt. Requirements need to be criticized, questioned and tested (even if only though thought experiments), because it is essential to get the definition of the problem as accurate as possible, and free from mis-interpretation. This precludes us from jumping into solution-mode too early. We need to take on the responsibility to shake down the requirements, and we should be among the harshest critics of the requirements. Only when requirements survive this test, and the scrutiny of the entire team and stakeholders should solution-space thinking begin.




Enterprise IT is broken (part 1)

Monday, November 18, 2013
posted by daveb

Broken windows in a St Petersburg abandoned cinema


You may have heard of Conways law:

“.. organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”

Conway’s law has proven to be true for every software project I have ever been involved in. Take the client-server application where the client was developed in C++ and the server in C. The client side developers were young and hip, and into OO. The server side developers were ex-mainframe developers in their 50’s. It’s fair to say the two parties did not see eye to about much when it came to software design. The friction between the parties came to a head in the shared communication libraries for client-server communication, which they co-developed. The libraries were littered with pre-processor definitions and macros that seemed to be at war with one another. The arguments between teams carried over into the version control comments. The shared communication libraries were some of the most un-maintainable, un-readable and bug ridden code imaginable, even though the purely client-side and purely server-side code were reasonably tidy on their own.

It was around 2008 when I began my adventures into this thing we call “the enterprise”. I was going to be an “enterprise developer” and take on development of a key part of the infrastructure – a credit card transaction processing interface. I understood that enterprise development meant you had to be lean – unlike a software company selling software products or software as a service, there were no economies of scale – you only build one instance of enterprise software.

As I began to find my way round some of the custom developed services and applications, a few questions started to emerge – like what version control system do you use here? Answer: “yes we have been meaning to get to that”. Ok, so there had been only one developer on that part of the project previously, and he was a bit green, so I decided I shouldn’t pre-judge the organization based on that alone.

More questions started to appear as my predecessor walked me through the operations side of things. He showed me how they had left an older version of the credit card processing API in production because there were an number of external clients using that interface, and they could not force them to upgrade to the new interface. Fair enough. I asked about where the source code was for the old version, in case I need to go back to it should a bug need to be fixed. Answer: “… well there shouldn’t really be any bugs, because it’s been there for years now”.

It turned out that work had started on “version 2″ without any kind of version control branch or label or even so much as a simple zip backup of “version 1″ source code. They had lost the intellectual property investment of the stable “version 1″, and had re-written large chunks of it to create “version 2″, which was not backward compatible, and was considerably less stable than the previous version. Unbelievable.

“Version 2″ had been 18 months in development, and had only very recently been deployed to production. Therefore, no business value had been delivered for 18 months. Business stakeholders had lost patience, and almost lost complete confidence in the development team.

Since the recent “version 2″ update, the phone had been ringing hot, and my predecessor would have an often lengthy discussion with an upset customer who had lost sales due to downtime and bugs with the service. I was now supposed to take these calls, and be an apologist for this mess.

At this point, things were looking so bad I was seriously considering walking out the door before I was even two weeks into the job.

However, I resolved to take on the project as a challenge, and that is the only thing that kept me there. I enquired about the testing approach: unit testing, integration testing, user acceptance testing and so on. In short:

Unit Testing: “what’s that exactly?”

Integration Testing: a spreadsheet containing a single sheet with dozens columns and a hundred or so rows, each representing a test scenario. It was un-printable, un-readable, inaccurate and was the un-loved product of something the boss had instructed the developers to do. The developers didn’t feel their job was to test the product, and instead of resolving this dispute with the boss, they had yielded to his pressure, but then done the worst possible job on it, to make the point that developers can’t test! This communication breakdown, and many other examples like it had almost eroded all value from the services being developed.

User Acceptance Testing: none

As we delved into architecture there were more surprises waiting for me. Like the messaging interface that used a database table as a queue, and had a service polling the table for new records every 500ms. This, I later discovered, would occasionally deadlock itself with another transaction and become the deadlock victim, meaning someone’s credit card transaction failed. The reason for using a table as a queue: the solution architect was a relational database guy and insisted this solution be adopted when the developers had hit some problems with the message queuing technology they were using.

Turns out there were more surprises in store

What is unbelievable is not that this dysfunctional situation could exist, but that project management and project stakeholders had no idea that these problems existed in the development practices and system architecture. They knew something was wrong, but lacked the ability to see any deeper inside the box. Nor did they have any notion that the industry has long since found best practices that address the very problems that were slowly destroying all value in the services they were providing.

At first I thought this organization was an anomaly, and that I would be unlikely to see anything this bad again. But then I started hearing about others who had seen similar things. And then I saw inside other organizations. I started to realize that what I’d seen was not an anomaly at all, it was practically commonplace. Sure, some were better than others, but I had yet to see inside an enterprise that had anything even remotely approaching a quality development team, with a solid set of practices that was able to deliver business value.

Conway’s law seemed to be holding true. Frictions between personalities and departments led directly to confusing and inconsistent software architectures. In fact Conway’s law can even be used in the reverse – where there exist strange inconstancies in a software architecture, you get an insight into the friction between different personalities or departments involved in its design.

If you want to assess your development team, and you aren’t a developer, just use the Joel Test. It goes like this:

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

Add one point for every “yes” answer and there’s your score. As Joel himself says:

” The truth is that most software organizations are running with a score of 2 or 3, and they need serious help, because companies like Microsoft run at 12 full-time.”