They Fit the Specifications, so What?

Some time ago, we made two test cycles with an Asia-based team, just for the hell of it. The first test cycle was given a very clearly written test specs based on the product’s specifications.

The testing was conducted in India and the results were as expected. 18 defects were found among 40 test cases – As many as had been found in the design phase already (*). The next test cycle’s mission briefing was clearly different. They had as much time for the testing as they had had the previous time.

For this assignment, they were only sent the product to be tested, and a list of things that are worth checking during the testing. That is to say, no specifications, nor test cases.

Test this product to your best knowledge and report all issues in a way you find best suited

As a result, we received 25 reports. To report the defects, the test team got creative and used video capturing to show them, so everyone could see what they were and how they happened. The software developers also gave thumbs up!

Of the time spend in executing testing work, the result received was 38% higher when the testers’ hands were not tied with having to closely follow specifications or test cases. The tester put on the end user hat and reported defects to the best of their experience.

The chasm in specifications between functional and genuinely functional in this case was pretty intense. The underlying problem in this case is that requirements engineering is conducted with almost no exceptions before building the product. The specifications are an abstract entity inside the designer’s head, formed prior to tell how the end product should function. Keep this in mind when you design the testing.

Remember that a functional product according to specifications might not be a genuinely functional product. Only when the testing has been done with an open mind can you tell how it really is!

(*) During the design phase of the testing we had to familiarize ourselves with both functional requirements and investigate the product itself, in other words: To conduct investigative testing. This design phase therefore took a lot of time.

, ,
Software testing

Pay It Forward:


  1. […] from making too accurate test cases. Take up, for example, checklists in test case scenarios, or become familiar with threaded process model. This way you save time from […]

Leave a comment