Bug Levels, or how simplicity made our life easier

Had a conversation with Jordan Sissel at devops days regarding log levels and the different understanding between dev and ops people. Jordan called this a language problem. I will write another post about that topic, but the discussion reminded me of another situation, where some confusion existed, due to unclear definition of levels. Here is the story.
 
Some years ago, we had more than 300 Bugs in our bugtracking system, about 50% of them older than 3 month and some even older than a year. It took us some effort to manage these bugs, and we were quite frustrated with this situation. Especially as the IT seemed to be the only department that cared about the bugs. Product management did not care, they more focussed on new features. 

Here are some of the things we did and what we achieved:

Many bugs were left unassigned for quite a long time, and sometimes bugs would ping pong back and forth between teams. To overcome that, we introduced a dedicated role to manage these bug, called bug dispatcher. This guy would try to find out which team was most probably responsible for fixing the bug and push them to fix it. He would among other things be involved in discussions with product owners if a given bug should be fixed now, or not. This helped a lot to bring new bugs to the right teams fast, but it did not really reduce the amount of open bugs a lot.

So next we decided to close all bugs older then 3 month. If they were not fixed within this time, they could not be important. But this also required some effort, as we were not consequent enough to just automatically close them, but we ran around, and talked with the PO to close them.

As this was not such a great succes, we had a look at our bug levels and the duration a bug would stay open depending on the level assigned. We had quite a lot of different bug levels in our tracking system at this time : 

• Blocker
• Critical
• Major
• Normal
• Minor
• Enhancement

As far as I remember, we basically took the levels the tool was shipped with. Nobody could explain, if and how the handling of a minor bug in relation to a major bug should differ. What time would be acceptable.

It showed up, that bugs having the first three levels assigned, would be fixed within a reasonable amount of time, all other levels were interpreted as: "we will never fix this".
    
Having understood that, we drastically changed the number of bug levels and the rules applied for handling these.

We now have only two bug levels left:

• Critical
• Normal

Critical means, we are or will be loosing money or reputation due to this bug, which means we fix it immediately if found in production, or we will not push this release to production if found on staging.

Normal means, we fix it with one of the next releases.

If the bug is in the tracking system, it will be fixed, unless the product owner excepts the bug behaviour as acceptable for the product, and closes the bug.

This reduced the number of open bugs drastically. And it also took away a lot of discussions. Some simple and easy to understand rules made solving this problem much easier. Getting rid of all the other levels enforced a decision.

Simplicity makes our live much easier now.

I want office desks with wheels

The one problem I remember from all IT retrospective meetings, many from back in times where DevOps was not yet practiced, is "communication", or more precisely "missing communication".

One of the principles behind the agile manifesto I think fits here is:

The most efficient and effective method of conveying information to and within a development
team is face-to-face conversation. 
Although I would replace development team with organization.
I think it is well known, that geographical distance affects communication. Somewhere (can't remember where) I read a hypothesis, that you basically have four classes of communication depending on distance: Same room, same building, same country, other country. And Between each class communication quality/rate/propability will drop significantly. Maybe by an order of magnitude?

Communiction needs people to overcome a wittingly or unwittingly present barrier. The more complicated communication is, the higher will be this barrier. Will you pick up the phone, will you move to check if the other one is at his desk or do you fall back to writing an email, which will make communication asynchronous, and is a one way communication prone to misunderstanding. 

If you are in the same room, you can communicate with everybody just by speeking up. Hopefully only people involved with the same product are in this room, and there are no (cubicle) walls.

If you are in the same building, You can stand up and walk to the person you need, but you may not find her and waste time.

In the same country, you will most probably use electronical communication options. 


Other countries may add timezone and language issues.  

In short, people should talk, face to face if possible. 

But what, if you are not in the same room, but in the same building. And your issue with somebody else is more than a short talk. Something you have to work on for let's say hours or even days together.

Well, I think you should sit together for this time. People/teams should relocate to optimize communication. And this should be as easy as possible. Great if you have Laptops and wifi, and there is some free desk space around. If not, why not using desks with wheels, so it is easy to move it around, plug in power and network, and off we go.

I want that for years, and did not get it yet. But now I read the Valve employee handbook. Wow, these guys, among other very interesting ideas, have desks with wheels, and an automatic tool to locate where people are sitting, depending on where there workstations are plugged in.

I like that idea. Especially as I think it is not a coincidence, that the word agile is a term related to motion / movement.

Let's move!  

Reading "The design of everday things" will change your life

Some month ago, I read a tweet from @oschoen, about the book "The Design of Everyday Things" from Donald A. Norman stating that reading this book (quote:) "will ruin your ability to walk through doors without complaining". I wondered why, and started reading.

Now some month later, after reading the book, I know what @oschoen meant.
Just visited the office of a law firm. Very stylish office design, but even people working there for some time do not know how to operate the doors of the wardrobe. And that is only one example. Doors that give no clues about how to use them, or even worse, give misleading cues. Taps that win design prices but until you know how they work, you always have to fiddle around with them for some time.

To make it worse, there are catastrophes that happen because of the bad design of switches or other things needed to operate power plants, trains, airplanes or whatever. And people feeling bad about themselfes not understanding how things work, instead of blaming the designer who built them.

Everybody designing products, mobile or PC applications and web interfaces should give this book a try.

Berlin city center, stupid speed limit of 10 km/h

For some time now, there is a speed limit of 10 km/h in on the minor streets around Berlin's Hackescher Markt, a famous place for Tourists, but also famous for art galleries and craftsmen.

Because my wife has her workshop around this area, I often have to drive through some of these streets. This is not much fun.

Why?
Well, this speed limit is completely useless. Nobody takes it seriously. If you drive 20 km/h you will get much fun. Especially if you have a cab or a parcel service driver behind you. If they can not go at least 50km/h they will start pushing you faster.

I could understand to limit the speed to 30km/h to reduce the amount of traffic and noise for people living there or visiting the area. But I do not see the local people stick to this 10km/h speed limit. Even bicycles would need to go slow, and of course nobody does that. But if you put a limit in place, then please also enforce it. Otherwise this teaches everybody to ignore trafic rules.

I never saw the police measuring around there. Many people would loose their driving license every day if they would. Or the limit would have been changed allready, as many people would have complained.
So please, Berlin City Administration, do something! And I would prefer to see a suitable speed limit.

PS: I just learned that police does measurements at the Gipsstrasse. So be carefull :-)

What I learned from John Allspaw and Eric Ries about root cause analysis

In his talk Advanced Postmortem Fu and Human Error 101 at the 2011 velocity conference, John Allspaw talked among other things about root cause analysis. One of his points was, that there is no such thing as a root cause for any given incident in complex systems. Its more like a coincidence of several things, that make a failure happen. I liked his visualization using several slides of Swiss cheese, where accidentally the holes of several slides of cheese are aligned in a way that a straight like can run through the holes, as a symbol for something bad happening.

In hindsight, there often seems to be a single action, that would have prevented the incident to happen. This one thing is searched for by managers when they do a root cause analysis. They hope to be able to prevent this incident from ever happening again. But if the incident was possible due to the coincidence of many events, it makes no sense to search for a singular root cause. This would lead to a false sense of security, as in a complex system there are to many ways, an incident can possibly happen.

Now I just recently read "Lean Startup" from Eric Ries. In one of the last chapters he suggested to use the five why method on incidents. So if something unexpected happens, ask why it happened. The next why is then applied to the answer of the previous question. First I thought, he is "only" looking for the root cause, and this would not make to much sense, as explained above. But there is the point that asking these questions will not only find an underlying cause, but better will bring to the light a chain of events leading to the incident. And Eric Ries recommends to find a counter measure on every level of the chain of events. This will make sure, that we are not just fixing the symptoms, but will improve the immune defense of our system.

I like that idea. It imposes much more work than only preventing the "root cause" but it gives a much better understanding of the system and is a good training for everybody.

dbdeploy with checksum validation

We just published a modified version of Graham Tackley's dbdeploy tool at github. Dbdeploy is enhanced with checksum validation of allready applied scripts. It will throw an Exception if an allready applied database script has changed afterwards.
Thanks to Michael Gruber for implementing this change while being a part of our Delivery Chain Taskforce.

database schema evolution followup

Having thought about the comment of pvblivs to my previous post, a substantial difference between version control of software and database changes became obvious to me. It is not a big deal, I just never thought about this before.

For software, the version control system (VCS)  represents a hopefully consistent state of your software for any revision. This state will be delivered to your dev/staging/production system. Maybe the software is build into an executable format, but basically it is taken as a whole. Maybe for optimization you only push the actuall differences to your systems, but still the final state of the target system will be represented by the revision in your VCS. If you want to know what has changed between to revisions, you ask your VCS what the differences are between these revisions, and it will show you.

With database changes, you actually write scripts that describe the transition between previous state and required state. So the db developer manually does the job that the VCS does for software. The VCS is basically used for storage and historization of these transitions, only.

In contrast to software, databases typically have two streams of evolution. Structural changes and Application data is pushed like software bottom up from development to production. The other stream is a top down stream of data added to the database by users. I think that this is the reason why we define changes and not the required target state as we have to find a way to merge the two streams.

A question I have now is, should there be a tool that automatically determines the required changes between two database states?

I remember listening to a presentation about a db migration tool, I think it was liquibase, you could point at two different databases and instruct it to migrate one to the state of the other. And I remember the bad feeling I had regarding that idea. Mainly because I did not like the idea to move changes from staging to productionthis way, because you would have to make damned sure not to accidentaly delete the production data. You would need to define very well, which application data to move and which user data to leave as it is. But maybe I should rethink that.

What do you think?