Arkisto: September 2010

Show Me a Good Test Case

22. Septemberta, 2010 | Kirjoittaja: Antti Niittyviita

Test cases are usually crap. They have been planned poorly and written even worse!

If you do not believe my claim, feel free to try it out yourself:

Make 100 test cases and give three different testers the same assignment of running them.

All three test cycles are very likely to give out different results. Running them takes every tester a different amount of time, and the pass/fail ratio in the tests differ from each other. I would be as bold as to wager that the number of reported bugs with each tester varies.

When one stops to think about it for a moment, the reason is obvious. The maker of these tests is usually someone else than the test runner. The danger lies in that only a marginal portion of test writers is actually good at programming people’s behavior. The depictions in test cases seldom succeed in giving each reader the same visual image of the test’s goals, background and methods of execution.

Planning accurate test cases is for this reason therefore very risky business. Going for accurate test case specs is often given reasons of accumulating good and comparable history information, especially from regression-type testing. Yet, the people who run these tests often switch and along with that, the test results. In a traditional specs-based testing process a whole lot of work time is consumed by checking the test results and normalizing them.

It can be reasonable way to spend time only when the project’s billing principle is hourly and the client has an endless supply of money.

Therefore I recommend thinking about the problem mentioned earlier with each project. The solution can be discovered with for example:

  1. Each tester runs only test cases they have planned themselves. Procedures made by themselves are usually understood the right way. This ends up saving time from checking test results and normalizing them.
  2. Refrain from making too accurate test cases. Take up, for example, checklists in test case scenarios, or become familiar with threaded process model. This way you save time from both test result checks and normalizing, but from test specs and the review.

Funny. At this moment, I don’t know what a good test case is. Do you?

Testing Under Schedule Pressure

9. Septemberta, 2010 | Kirjoittaja: Antti Niittyviita

Testing-guru Amit has started working in an enterprise, which develops logistics systems for industrial businesses. System development is project-based, because the needs strongly vary between different clients.

Working is stigmatized by continual hurry and having to be at several places at once in the product development. When nearing the agreed upon systems delivery, the work piles up heavily. Usually, the last month before delivery the entire team has to work overtime. After the delivery, time is spent to fix bugs that remained in the system, which the client has made reclamations about.

When Amit came on board, the enterprise did not have an actual testing team – they subcontracted testing from a third party. The modus operandi for testing was, that product development made a testing assignment to the subcontractor whenever they felt like it. The deal was that they paid a fixed amount for each test cycle.

Amit quickly took note that the product development was plagued by a nasty bottleneck: Amit thought that having to be in several places at once and scheduling problems were caused by the sheer number of critical bugs discovered near the end of the development cycle. Fixing those bugs took a significant chunk of the time they had reserved for making the finishing touches on the systems delivery. In addition, the project management had to make tough decisions prioritizing which bugs could be left unfixed, and which ones had to be fixed as soon as possible. The remaining bugs were also the ones which the client made usually reclamations for during the warranty period, which in turn consumed even more time when it came to fixing them.

Fortunately, Amit had once seen Conformiq’s set of slides about subcontracting testing services. They included an excellent illustration of the schedule pressure.

The project manager did not call for testing until the last third of each delivery. The manager thought that it was not worth it to start testing, because the budget included a fixed number of test cycles and he knew that there were flaws in the software. Starting the testing cycles on time felt like a waste of resources to him. On the other hand, systems delivery had a deadline, which meant that there was a contractual penalty if they did not meet it. Therefore, they had to leave defects for the warranty period. In addition, a big chunk of time for the last third of the testing assignment was spent verifying the reported defects and checking up the fixes.

Luckily, Amit, being the old guru he was, had experienced it before. The solution was plain obvious. No one had yet examined the problem field from the perspective of testing.

Amit suggested that the systems delivery was to be phased clearly to testable builds for the entire length of the project and that the testing budget was to be divided among the entire life cycle of the project. These adjustments to the process model as suggested by Amit caused the following actions to take place in the enterprise:

  • Build cycle was condensed to two weeks. Each functional build had to be ready on Thursdays of even weeks.
  • Each build was to be given as a testing assignment for the testing. Test results were ready in five days.
  • Error fixing process was started immediately when the first test results arrived, and they were continued until the end of the project.

In addition, they agreed upon a fixed monthly price with the testing contractor, which included two testable builds. This price was billed if there was or wasn’t work. Anticipatable workload reduced the price for test cycles. Merciless billing policy ensured that product development took care of regular, evaluable, builds.

These measures caught critical errors earlier and they were fixed in good time before delivery. Workload of product development remained steady throughout the project. There was no overtime to be made and the number of fixes during warranty period was halved. The enterprise noted that:

Both testing and coding are governed by the same rule: They do not work optimally under schedule pressure!