Arkisto: August 2012



The Famous last words of a CEO

30. Augustta, 2012 | Kirjoittaja: Antti Niittyviita

I was woken up in the night by my own scream!

Ohjelmistotestaus.fi had received an unexpected boom in visitors. During one day, our site had 33,000 readers. This number was over 10 times the normal, as recorded during the last night. I was quite content, since this was something we had secretly wished for since starting the blog. We had invested a massive amount of work hours into it as well!

Our testing guru Esa asked me during that first morning a question which cooled my coffee into its cup and stopped the clock in our office wall to the time of 09:54. My vision dimmed.

Can the blog handle this kind of stress? Has it been tested?

At the very same time my mind was flooded with horror stories from almost successful success stories and of terrible catastrophes.

  1. Sampo bank’s web bank renovation which was buckling immediately after the service break and caused customers to vote with their feet.
  2. VR’s ticket sales renovation is still being discussed, even though the customer numbers did not even double their number from the expected.
  3. Fortum’s error notification system that crashed during the blizzards of the previous winter. Customers were stranded without electricity and without the supplier’s support service.
  4. Marimekko managed to get their logo to the front page of Google search engine as a part of a marvelous PR trick. This entire PR potential became for naught as their page crashed in under an hour.

All this ran through my mind in about 30 seconds. As the clock hit 09:55 I managed to just barely compose myself. I coolly and in blind idiocy replied to Esa.

I suppose it will hold. The blog is maybe pretty light and our service provider is allegedly pretty good.

Everyone with common sense would see the idiocy of my reply immediately and might be giving a little smile as well. I blatantly lied to Esa and myself. Luckily, those words I just uttered did not remain the final words of this story.

An opportunity for an explosive success usually comes but once. If your service cannot hold during that moment, the customers are unlikely to return. Do not burn your software development’s and marketing’s investments by neglecting the simpler tests.

Stories from the Wonderland of IT Projects

21. Augustta, 2012 | Kirjoittaja: Antti Niittyviita

Our reader has approached us with a very interesting story regarding public sector’s IT projects. The story is sure to rouse feelings for and against.

Hi,

Along the years I have read a couple of stories from your blog that have to do with public IT projects. I would like to shed some more light into this subject.

I was recently a part of a largish IT project in the capital region. What made this special was that the Client had acquired professional testers to secure the quality of their project. I was one of the testers the Client had gathered.

The project had to do with a system that ran in an SAP-environment. The system had several parts, each of which had their own Supplier and all the Suppliers had to have their parts talking with the parts of other Suppliers.

Well, we naturally started the testing work as soon as we started. The Suppliers had already managed to build the system so far as to enable logging into the individual parts of the system and test them separately. It only took a few hours from the start of the testing to make us realize that what we were looking at could become the new Finnish record – or even the World record – of writing bugs. I emphasize that we had only just started testing for the individual system parts of the Supplier. We found bugs from all the different areas of software testing, starting from usability and going all the way to security.

We reported the bugs immediately and when the report reached the Suppliers a show started, one like I have never seen before in my career. The Suppliers took the defensive immediately and started a trench warfare against the testing.

“We did not know how to test, or we tested against the process and we tested wrong things.”

One representative of SAP-Supplier yelled straight at us with a red face, since we had found too many bugs. They did not want to fix them, at least not with the same price.

This testing ‘against the process’ became my own favourite. The Suppliers thought that when they had agreed upon that the order of button presses in the process were first 1 and then 2 that no other orders were acceptable. We, of course, tested what would happen if one were to press 2 before 1 and lo and behold, we discovered that in far too many places the system dissented and finally crashed.

It is senseless to presume that the end user would never press the buttons in the wrong order.

It started to unravel itself to me, piece by piece, what was going on. The Suppliers had grown accustomed over the years for the bug fixing to be billed separately and it had formed a significant part of the project’s billing. It is good business for the Supplier if the IT project were to delay, since it would mean bigger billing for them – and believe me when I say that the billing did run.

I occasionally joked with my colleague that the billing must be the single thing of the Supplier which works without a hitch. I ended up with an understanding that this was the first time, especially for the SAP-Suppliers, when they encountered a scenario where the Client had the order tested by outside professional before accepting the order and that this course of action jeopardized the Suppliers’ normal billing routines. I personally explained to the Client that it is not worth paying anything for a delivery that was this buggy and before they can even consider paying the bill, the bugs have to be fixed and through testing, be confirmed as such.

We had the chance to serve as the fire wall for the Client for half a year until one day we received a notification:

The Board of Directors had decided to fire the Client’s testing team.

Well, that is that, I thought. We had already managed to get a lot of things advance and we had taught the Client important things regarding IT project ordering processes and software testing. I was quite content despite everything, but at the same time I understood that this project would not be finished for years even though they were quite close already.

But this was not the entire story. While I still was in the project I heard that the Board of Directors quite handily consists of the representatives from the Client as well as the Supplier. I agree that it is a good thing that there is a common forum for these people, but this Board can decide, for example, whether or not they’ll remove the testing from the Client.

I made the decision of writing this story only after I heard from another testing colleague that in his public sector’s IT project the Board of Directors had the same thing going on. They had straight discarded most of whatever testing results had come in. That particular IT project failed badly and we got to read about it in the country’s number one news media.

This story could continue for ages and we would not see a shortage of juicy details. Finally, I should mention one of those details. We, in fact, encountered a situation where the Suppliers would have wanted to sell the Client their own testers – or ‘consults’, like the custom is to say in SAP-circles. As an idea, this is quite incomprehensible.

The situation would be exactly the same as you would service your own car and then perform the check up for it with your own hands.

Despite everything, that particular project must have been the best I have had in my career. I had not seen since my armed service when a grown up yells and wails with full force during working hours. I got to experience concretically the power of testing and I needed this experience. This is a good place to continue onwards.

When the Supplier yells with full force, the tester might have done something right.

Shoot First, Ask Later

14. Augustta, 2012 | Kirjoittaja: Antti Niittyviita

I visited a theme park this summer. Every theme park’s essential equipment includes a classic hammering attraction. My mind was vividly reminded by the previous TalentIT seminar that was held in Aalto university. Reaktor’s stand had a nice party going on. People were queuing to take a try at the seminar’s number one attraction, the classic “High Striker” game.

I followed the game and the attempts of people with interest. The people studying IT and the people working on the IT sector were keen to land their strongest blows, but very few of them observed various techniques from the side or stopped to think how to reach the best result.

To put it in more familiar terms: They shot first… and did not even ask later.

It is regrettably often that testers go and blunder in the wonderland of exploratory testing the same way. They act as if having lived in a barn by eating bugs, but none of them actually try to understand what is going under the hood. After all, that is arduous and makes you think too much.

Despite this, without exception the most useful results produced by testing are born from thinking for a moment before acting. By seeking understanding.

In High Striker the best result could have been achieved by taking a look at the game mechanics from the side. In this game, one hits the other end of a swing-like board. On its other end, a weight jumps up due to the strength of the blow. The game measures how high the weight jumps and told the ‘strength’ of the hitter this way.

The weight is not very heavy, so you don’t even need a large momentum to make it jump. Therefore, the sensible approach would have been to hit the board near its center and holding the very far end of the maul’s shaft, so that the swing would be as fast as possible and that the weight would has shown quite decent results for all swings.

Despite this, everyone hit the end of the board and the results reflected this.

Suddenly, a question arises: What does exploratory testing really mean? To me, it means asking first, shooting later.

Therefore, a tester ought to stop frequently. To explore the product’s architecture, concept, and functionality. Only this way can one understand the whole and finally produce the most useful results from testing.

A Nominee for the Most Expensive Bug of Year 2012

7. Augustta, 2012 | Kirjoittaja: Antti Niittyviita

The summer holiday is soon to end, the nights are getting darker and the working schedule is becoming tighter.

The blog opens with a light bug-related news. This blog has treaded upon very costly bugs from time to time. The competition for the year 2012 find itself presented by a compelling newcomer.

A Jersey City based Knight Capital’s trading software algorithm had a defect – a bug had managed to slip in unnoticed. The bug had caused damages worth 440 million dollars. The next day, the company’s value plummeted by over 50%. By the end of the day, the stock value was around one third of what it used to be before the bug.

And that is not all. When you browse Knight Capital’s website, the actual damage can be significantly more severe. The company is marketing their technology as a core of efficient and profitable business. The technology is mentioned on the site along the lines of:

Our robust technology infrastructure and flexible architecture provides instant access across multiple asset classes to liquidity both within and outside of Knight.

And:

With our regular, prudent investments in our technology platform, Knight strives to remain ahead of the latest technology developments, providing advanced market access and trade execution services across multiple asset classes.

What do you think the current and future clients think of the company’s credibility when their vanguard technology fails in such a dramatic way? One thing is for sure, it is not going to increase the trust of clients. I would not be surprised at all if the investors would start to withdraw their funds. At least the comments in the news foretell a dire future for the company.

Should you market your company’s technological superioirity, you inevitably have to put in efforts to guarantee that it works. Otherwise, bugs become way more expensive than their real value.