Avainsanat: ‘cost of quality’

Even Insight has its Price

17. Februaryta, 2015 | Kirjoittaja: Antti Niittyviita

If we are not talking about unbelievable luck, chance or fortune, then even successes have their price.

It is possible to pay this price beforehand and in that case, the success has been earned. Or, you can pay it afterwards and then it comes with interest.

It is simple to give the matter some thought, let’s say with regards to the upcoming winter holiday:

Which one feels better? To make a holiday trip to the sunny south during winter with hard earned savings. Or do it one swipe of a credit card and then keep on paying those credit payments with interest even until summer is right on the doorstep?

If your company’s values include flexibility or client-orientation, then would it be reasonable to ask what kind of price are we paying for those? What sacrifice are we willing to make each day, for example, on the altar of client-orientation?

Being fit has its price, too. It is called sweating. Likewise, it is difficult to purchase insight, unless you are first willing to withstand confusion, or mental efforts.

Even successes of software projects carry their price. It is often unsurpassably heavy to acknowledge the imperfection of your own software development. Even though you were to regularly meditate in retrospect in lotus position.

The price that comes with interest can be recognized from the moment when ‘stuff’ hits the fan.

British Airways Flight 38

8. Januaryta, 2015 | Kirjoittaja: Antti Niittyviita

In January 2008, flight captain Peter Burkill, with his crew, welcomed the passengers aboard to the flight from Peking to London. The route was one of his favourites, since it was a day flight. On the 10 hours long flight one could see the scenery change from Mongolian plains to Siberia and Scandinavia.

The trip on Boeing 777 went smoothly, like any intercontinental flight. It was a relaxed routine from by an experienced flight crew.

But, 3.2 kilometres before Heathrow airport the crew noticed that something was amiss. The autopilot was supposed to slow the plane’s velocity down by adjusting the angle of the flaps and by at the same time increasing the output of the engines. Yet, neither of the plane’s two engines responded to the attempts of the automatics to accelerate to ensure a safe landing. Despite their best efforts, the two pilots could not return power to the engines either.

At the altitude of 46 metres, the co-pilot John Coward took the plane over with the manual controls, while captain Burkill adjusted the flaps in an attempt to lengthen the landing glide. The plane barely passed over the crowded trunk road A30 and the southern maintenance way of the airport as the captain confirmed their Mayday situation to the airport control. The landing gear touched down on the ground for the first time at 270 metres before the beginning of the runway 27L on the lawn of the air port area.

During that impact, the plane lost its front tires first. Next, as the main tires were torn off, the right tire punctured through the fuel tank and the fuselage into the cabin. The left tire only went through the wing. The plane turned to face the runway one side first, and eventually the speed stopped right at the beginning of the asphalt.

As if a miracle had occurred, all of the 136 passengers and the entire flight crew survived the landing. Familiar movie-like special effects were avoided altogether. No explosions, no balls of fire. No blood, no guts. The passengers were safely extracted from the plane in a matter of seconds, and the rescue vehicles were on the scene immediately. The passengers of BA38 narrowly avoided a gigantic catastrophe, but how did the flight crew know how to act the right way during the crisis?

It is not a happenstance that flying has become the safest form of travel. It is no accident that with Boeing 777 planes alone there have been over two million flights before the first accident. In air travel, the staff knows how to act without a moment of hesitation and just the way that has been agreed upon. The same holds true for both matters of routine and for times of crisis. And what is most important, it is likely that human errors occur a lot less frequently there than in other forms of business.

The answer is remarkably simple. One of the centermost tools of ensuring the safety of air traffic is the checklists.

Boeing even has an entire unit, whose job it is to chew through the hundreds of pages of accident reports and then process them into a useful form. Checklist experts like Dan Boorman work there.

I use seatbelts even though I know how to drive a car. I also use life jackets, even though I know how to swim. I used to oppose checklists earlier with a feeling. After all, I know what I am doing. Dammit! Then I happened to read Atul Gawande’s blinding book The Checklist Manifesto. It drills into the undisputable benefits of checklists from the perspective of medical doctors and surgeons. Finally, I had to admit the truth.

A checklist in a critical part in even my projects can be like a seat belt in a car, or a life jacket on a boat. It can be the factor that separates a success story from a catastrophe.

You Can Take Too Much of Even Superfoods

25. Septemberta, 2014 | Kirjoittaja: Antti Niittyviita

My temples were under merciless pressure. Light cut my eyes and cold sweat broke. There was a sensation of vertigo from relentless headache and I felt like throwing up. I suffered a powerful migraine attack yesterday.

Luckily, I belong to the group of people who get better from a Burana tab and a little bit of coffee. Increasing the dosage of those two remedies makes me feel better faster and clearer, but how much is enough? If I take 5x800mg of Burana during the morning hours and five cups of coffee, then what?

Excess breeds nausea. Or gives a gastric ulcer.

Superfoods, pain killers, stimulants, alcohol, fitness, resting and even testing become detrimental when taken to extreme amounts. The benefit-graph stands true for pretty much all kinds of activity.

By increasing the effort, you gain more benefits… until the effort becomes excessive.

Optimal level is not discovered by guessing the dosage when one is already in a hurry. It is discovered by starting at once and regulating it more accurately with experience and learning.

If A Pelican Goes Through The Turbine

20. Mayta, 2014 | Kirjoittaja: Antti Niittyviita

U-2 is a spy plane developed in the 1950s by Lockheed. Originally, it was developed especially for spying on the Soviet Union during the cold war. What was special in the plane, was its light structure and for that time and age, the high maximum altitude it could reach. One could reach even 100,000 feet (~30,5 kilometres) high.

An altitude that high is not typical for our more modern planes. Los Angeles airport learned this the hard way.

The air control system was not designed to handle U-2’s flight altitude and the system malfunctioned.

The system interpreted, due to a simple software bug, the spy plane’s flight altitude to be very low, which caused a forced chain reaction in the system for readjusting the flight routes for every other plane. The system could not handle the stress and all the flight traffic control in southern California was left paralyzed for an hour.

Merely at the Los Angeles airport 27 flights had to be cancelled, and 27 other flights were rerouted to other airports, while 212 flights were dramatically delayed.

For example, one flight from L.A. to New York costs the air company $56,400.

Simply put, total expenses caused by the cancelled flights can reach even $1,500,000, and that is without its follow up effects.

With that kind of money, an expensive ($150/h) test consultant would work for 10,000 hours. That’s 1350 working days!

Sense of Security that is but a Mirage

18. Februaryta, 2013 | Kirjoittaja: Antti Niittyviita

Savonsanomat.fi was down due to people’s thirst for knowledge brought about by the school threats last week. The main goal of Savon Sanomat, like with any other news service, is to get clicks. Their business is dependant on the user traffic. Sometimes people make gigantic marketing campaigns to reach this goal. Occasionally, some idiot takes care of it for the media houses by, for example, presenting new fake boobies or by making a bomb threat against a school.

Every chief of a web news secretly wishes in the back room that this time around their own news piece would become a hit and that the user traffic would explode. Which is why the headlines in web news are click baits.

What is baffling, however, is how poorly most of these web services have prepared for these success stories. And this is not even dependant on their field of business.

In all the most popular web services, sudden spikes of user traffic has been taken care of beforehand. Furthermore, their service is being stress tested constantly in case of a hostile environment.

Google employees test the contingencies of their service centers regularly by making attacks against them. In these attacks, they do, for example, information network disturbances or drill holes into the pipes that contain the coolant. Similarly, the web service Netflix is also under constant challenge. The Chaos Monkey tool of Netflix randomly attacks their servers 1000 times a week.

Excuses for neglect are found in hundreds behind the office tables. Testing is so painstaking. Surely it must cost too much money and time. Besides, the service has worked well so far. I can assure you that none of these are true. The only reason for neglect is limited ability to get it done.

Every person with common sense knows that practice makes perfect. Yet, when it comes to web services, they neglect the practice. Because everything has worked so far, they fall into a false sense of security. They fall for a mirage.

Avoiding Responsibility, or Ignorance?

30. Januaryta, 2013 | Kirjoittaja: Antti Niittyviita

Web services are made to be cooler and cooler. They successfully respond to the increasing variety of terminals. Top products are launched onto the market on a daily basis. Each one has links to social media, and a user interface that is sheer deliciousness. Be it for either business or consumer clients, people make efforts to produce good functionality in web services and for good user experiences.

Quite often the product owners order works from the experts of software development and they are also taking care of testing. They are whistling past the graveyard.

Despite all that, there remains nearly always a black hole in building web services. Behind its event horizon there is something, which no one knows inkling about.

The non-functional areas of web services, such as information security and performance have a central role to play in building a successful web service. Despite this, the responsibility of taking care of them almost always remains in a grey zone. The question remains awkwardly hanging in the air like a dark rain cloud whenever a third party testing partner sits at a meeting and opens his or her mouth.

The client is responsible for the new product’s ability to fill in the criteria of the business. The developer is responsible for making sure that these criteria are met. The service provider running behind the scenes is the same which they have always used. Everyone thinks they are satisfied.

Yet, for example, the response times of a service has been noted to be a key factor for the money streams in a web business:

It is in everyone’s best interest, without an exception, that the provided service works. Regardless of what role you are in, be it the service provider, client, supplier or the tester, remember to bring this up: Who is responsible?! Answer it!

Urgency, what a Wonderful Excuse

17. Decemberta, 2012 | Kirjoittaja: Antti Niittyviita

It is easy to hide behind urgency. Especially when the urgency is real. Urgency is an attractive excuse to postpone something unpleasant indefinitely. Who would actually want to know what the status of the software is, if that will just cause extra work?

No can do, we’re far too busy. Let’s test in, say… March.

Urgency can be caused by several factors in a software project. Pretty often the project risks are about lack of resources or running out of time.

  • It feels like deadline is upon us and that there are too many issues still
  • It feels like the pace of development is hurried and torn
  • It feels like there is not enough manpower in the project.
  • It feels like the marketing has promised the impossible

Projects often start steady and peacefully. Then the pace gets quicker and finally the urgency grabs a hold of you. The last weeks of the project are fueled by kebab, coffee and sleep deprivation. That if anything is healthy. In addition, it feels good to be able to brag to your friends about being busy.

These problems are most typical in projects where the testing has been taken care of amidst your own work. Testing is done when you have the time for it and sometimes you can skip testing. Urgency is a wonderful excuse.

As a matter of fact, neglected testing due to sense of urgency is the main reason for the urgency getting even more urgent. Controlling problem spots, emergency updates and warranty fixes make deadlines impossible to meet. And then, you run out of resources.

Sometimes you are so busy rowing that you do not have the time to start the boat’s motor. That is when it is a good time to roll up your sleeves and get out of your trenches. Stop hiding behind urgency!

A Monologue Backlashes in Product Development

25. Septemberta, 2012 | Kirjoittaja: Antti Niittyviita

Last week we discussed a project that was already a month late from its schedule. Since money governs the businesses and testing has to be an investment, we decided to find out what was going on: We interviewed the project group.

Product development cycle in the business was 3 months in entirety and testing was emphasized in the end stretches of the cycle. After the interview we could state that the number of bugs found during the last stretches of the cycle had, like always, caught the software development by surprise. However, the number of bugs was not their greatest challenge. Fixing the problems just took a confusingly large number of time. Why was this?

Let us think about the situation where Calvin the Coder builds a new feature. The number of lines can easily number in hundreds. Then Calvin’s code from August-September reached the testing phase in October and bug reports start to emerge the fixing process can start. But what do you think? Can Calvin remember what kind of changes was done to the code in August?

Is it a wonder at all that digging through old lines of code can take so much time?

We stated that the same problem systematically plagued projects whose product development cycles were long and where testing was emphasized on the last stretches. First came the coding monologue, then came the testing. The product development was and stayed sticky. We figured out two possible solutions to the problem:

  1. Speed feedback cycle by forcing developers and testers to a dialogue
  2. Budget the project a plenty of time to fix things and hope for the best

You do not have to be a rocket surgeon to guess which option the product development lead chose.

Product development is always a team game. Quality includes staying in schedule. Only a fast dialogue between testing and development can guarantee fast results.

But it isn’t Worth in Investing in Schedule!

21. Septemberta, 2012 | Kirjoittaja: Antti Niittyviita

We dug our claws into a project, whose delivery day had been postponed by bugs found at the last minute for over a month already. Nothing new under the sun, we stated. This software development cycle had already been done for.

During the lunch break of the starting day we stopped to talk about money, like we have a habit of often doing.

When a project is delayed, what does it mean from the perspective of the project’s turnover or budget?

If the product development team takes 15 people’s full workload in a month, then a careful estimate would place the extra month of fixing bugs as costing dearly for their employer.

According to Palkkavertailu.com service, a software developer’s average salary is around 3600 euro. With the employer’s side expenses and holiday salary the entire expense comes to be around 5400 euro. A team of 15 people would therefore cost around 81000 euro in one month. (*)

The direct expense caused by a delayed project is rather high. The budget cannot hold and profit margins get swept even though the project could be a long one.

The delay costs not only money. The client can wait for the update with eager anticipation. The feeling of disappointment is pretty difficult to heal. In future, potential deals can be postponed, even remain entirely unborn. The resources allocation of the business gets disturbed, since the next project does not gain the people it was intended to get. In the end, even that project gets delayed. What we have then is a running start. Effects flood into my mind, when I stop to think about it for but a moment. Suddenly, a question springs into my mind.

How much would you be willing to invest for a project to remain in schedule?

(*) In reality, the expense is even greater when you consider work benefits, health care and tools!

Stories from the Wonderland of IT Projects

21. Augustta, 2012 | Kirjoittaja: Antti Niittyviita

Our reader has approached us with a very interesting story regarding public sector’s IT projects. The story is sure to rouse feelings for and against.


Along the years I have read a couple of stories from your blog that have to do with public IT projects. I would like to shed some more light into this subject.

I was recently a part of a largish IT project in the capital region. What made this special was that the Client had acquired professional testers to secure the quality of their project. I was one of the testers the Client had gathered.

The project had to do with a system that ran in an SAP-environment. The system had several parts, each of which had their own Supplier and all the Suppliers had to have their parts talking with the parts of other Suppliers.

Well, we naturally started the testing work as soon as we started. The Suppliers had already managed to build the system so far as to enable logging into the individual parts of the system and test them separately. It only took a few hours from the start of the testing to make us realize that what we were looking at could become the new Finnish record – or even the World record – of writing bugs. I emphasize that we had only just started testing for the individual system parts of the Supplier. We found bugs from all the different areas of software testing, starting from usability and going all the way to security.

We reported the bugs immediately and when the report reached the Suppliers a show started, one like I have never seen before in my career. The Suppliers took the defensive immediately and started a trench warfare against the testing.

“We did not know how to test, or we tested against the process and we tested wrong things.”

One representative of SAP-Supplier yelled straight at us with a red face, since we had found too many bugs. They did not want to fix them, at least not with the same price.

This testing ‘against the process’ became my own favourite. The Suppliers thought that when they had agreed upon that the order of button presses in the process were first 1 and then 2 that no other orders were acceptable. We, of course, tested what would happen if one were to press 2 before 1 and lo and behold, we discovered that in far too many places the system dissented and finally crashed.

It is senseless to presume that the end user would never press the buttons in the wrong order.

It started to unravel itself to me, piece by piece, what was going on. The Suppliers had grown accustomed over the years for the bug fixing to be billed separately and it had formed a significant part of the project’s billing. It is good business for the Supplier if the IT project were to delay, since it would mean bigger billing for them – and believe me when I say that the billing did run.

I occasionally joked with my colleague that the billing must be the single thing of the Supplier which works without a hitch. I ended up with an understanding that this was the first time, especially for the SAP-Suppliers, when they encountered a scenario where the Client had the order tested by outside professional before accepting the order and that this course of action jeopardized the Suppliers’ normal billing routines. I personally explained to the Client that it is not worth paying anything for a delivery that was this buggy and before they can even consider paying the bill, the bugs have to be fixed and through testing, be confirmed as such.

We had the chance to serve as the fire wall for the Client for half a year until one day we received a notification:

The Board of Directors had decided to fire the Client’s testing team.

Well, that is that, I thought. We had already managed to get a lot of things advance and we had taught the Client important things regarding IT project ordering processes and software testing. I was quite content despite everything, but at the same time I understood that this project would not be finished for years even though they were quite close already.

But this was not the entire story. While I still was in the project I heard that the Board of Directors quite handily consists of the representatives from the Client as well as the Supplier. I agree that it is a good thing that there is a common forum for these people, but this Board can decide, for example, whether or not they’ll remove the testing from the Client.

I made the decision of writing this story only after I heard from another testing colleague that in his public sector’s IT project the Board of Directors had the same thing going on. They had straight discarded most of whatever testing results had come in. That particular IT project failed badly and we got to read about it in the country’s number one news media.

This story could continue for ages and we would not see a shortage of juicy details. Finally, I should mention one of those details. We, in fact, encountered a situation where the Suppliers would have wanted to sell the Client their own testers – or ‘consults’, like the custom is to say in SAP-circles. As an idea, this is quite incomprehensible.

The situation would be exactly the same as you would service your own car and then perform the check up for it with your own hands.

Despite everything, that particular project must have been the best I have had in my career. I had not seen since my armed service when a grown up yells and wails with full force during working hours. I got to experience concretically the power of testing and I needed this experience. This is a good place to continue onwards.

When the Supplier yells with full force, the tester might have done something right.