Myth 1 – Testing is a one-off activity to demonstrate the application is error-free.

There are actually two misconceptions here:  testing is neither discrete, nor can it ever demonstrate that an application is free of defects. The two go hand-in-hand: taking the latter point first, testing can, at best, demonstrate that the way in which we use a particular piece of software to achieve our goals, performs in the way we expect. In order to do this, we need to be clear about what it is we expect it to do, and we can then reproduce these conditions in a “safe” sandbox. Which takes us to the first part: the documentation of our business processes, and the interaction with the software is not always well documented or readily available. This means that the testing phase is often the first time we collate in one place our expectations of the application’s behaviour. If I’d had a pound for every time I have encountered the phrase “it needs to work in the same way as the old system”, I would probably be on a tropical beach rather than in my home office! A description of the testing conditions and scenarios should be a living document, linked to project development and change control, so that when our processes evolve, our test cases follow suit, ready for the next round of testing. In practice the development of detailed test cases is usually done as part of the testing phase, putting at risk the critical “coverage” concept (the degree to which test cases adequately reflect the entirety of our operations). 

Myth 2 – User Acceptance Testing requires detailed step-by-step scripts.

As ever, testing is a “horses for courses” activity. There are times when exact step by step instructions and expected results are defined in great detail, and others when a simple one-liner is adequate. This needs to reflect the expectations of the target audience. I have seen examples where a script simply says: “Create and approve a timesheet” and others that have been accompanied by ten pages of screenshots.  The important point is that the result should be unequivocal. The level of detail often depends on the type of project that is involved. Which conveniently takes me to a corollary of this myth: “often the perfect is the enemy of the good”. In 1990 NASA launched the Hubble space telescope. Two months later a “spherical aberration” was discovered on its main mirror (in plain English, it needed glasses). A full optical test before launch would have cost $100M, but was not carried out, although a much simpler test could also have identified the flaw, had it been carried out.  

Myth 3 – Testing is a technical activity that IT can perform.

As with all good myths, this has some basis in truth. IT often are the de-facto experts and are usually able to carry out basic “skim testing” that can weed-out basic configuration errors before more experienced testers can get to work. This is both time-effective and provides a level of “four-eye” testing that is advisable. But ultimately only someone who can adequately represent the end-user can assess objectively if a product meets the expectations.

Myth 4 – Testing can be carried out by anyone.

Like with the previous myth, there is a kernel of truth, in that once a script has been defined, and any prerequisites documented, the actual tasks of carrying out the test can be carried out by anyone with the requisite access. This is not the whole story however and would miss out on the “off piste” element of testing. A tester should be free to explore areas at the fringes of the test. An experienced user will recognise (and test) exceptions to the normal “happy path” sequence that the test designer envisaged. It is often these meandering unscripted variations on the original test script that test the product better than any formal script could. A tester should first and foremost be knowledgeable about the process involved. I have seen teams of testers parachuted into a project to execute hundreds of scripts, who missed some obvious aspects because they lacked the real-world context that was needed. 

Myth 5 – Testing is expensive.

This myth has equivalents in many walks of life and reminds me of the adage by oil well fire-fighter Red Adair: “If you think it’s expensive to hire a professional, wait until you hire an amateur”.   While testing is an activity that requires an intimate understanding of your own internal business processes, there is a level of expertise that your partners can bring to bear in designing and managing this phase, and at Solitaire Consulting we have several decades’ worth of collective experience in this field and would be happy to discuss your specific requirements and challenges. This topic will be the subject of one of our “Lunch and Learn” sessions on 26th February, where we will look at testing strategies, governance, tools, and processes used in testing:  if this is an area that interests you, please look out for further details!

Myth 6 – The product is fit for purpose:  it is used by dozens of other customers.

This can be a difficult one, in that vendors (and customers) often genuinely believe that if a product has been in the market for a number of years, the bulk of its defects have been ironed out. This is possibly (and hopefully) true, but it is best not to assume that others will use it in the same way as us. The mantra of user acceptance testing can best be described in the words of Ronald Reagan, when discussing the process involved in nuclear disarmament treaties: “trust but verify”.