Exploratory Testing vs Scripted Testing: Who Wins?

Real-world benefits of exploratory testing:

Traditionally, software testing has been a very rigid activity, but in recent years there’s been a shift away from script-based testing. Exploratory testing, which is more context-driven, has come to the fore. That’s because it gives testers more freedom to exploit their skills and knowledge, and it makes them responsible for optimizing the value of their own work. 

Exploratory Testing vs Scripted Testing

Not everyone is sold on the value of exploratory testing. The perceived lack of formality and emphasis on personal responsibility can set alarm bells ringing. But that concern is largely based on a misinterpretation of exploratory testing. It’s not about throwing rules out the window and testing at random, it’s actually very structured and systematic. And it’s also highly effective. 

Skeptics want concrete proof that it does more than improving tester’s morale. That’s why we decided to conduct a study that would pit context-based, exploratory testing directly against a script-based testing approach. The results were very interesting, as you’re about to find out.

Context-based (Exploratory Testing) vs Scripted Testing Teams

Two teams, two approaches:

We started by dividing the testers into two teams of three. Testers in each team had the same comparable application knowledge. The same definitions for defect severity (major, minor) were established for both teams. Both teams had the same application build delivered to them. One team (“scripted”) would apply a traditional script-based testing approach and the other team (“exploratory”) would adopt a context-driven testing approach. The testing activities would be divided into two phases of three days each.

The script-based team identified five business work-flows to test and generated 15 test cases. The test cases were limited in scope, so testers didn’t have any freedom to explore outside the confines of the script.

The exploratory team created two visual mind maps, one that identified the test coverage and test charters, and the other covering product components/modules. The process produced 24 test charters in total. The charters defined were high-level and allowed contextual interpretation, extending the scope of the test session for the testers.

Phase 1:

The scripted team managed to complete 6 test cases in the three days allotted. They reported 6 major defects in that time.

The exploratory team managed to complete 13 test sessions ranging from 30 minutes to 180 minutes each. They reported 10 major defects and 5 minor defects.

scripted vs exploratory testing 1


Interestingly the exploratory team reported all of the defects that the scripted team had reported.

Phase 2:

The scripted team managed to complete 9 test cases this time. They reported 10 major defects and 8 minor defects.

The exploratory team completed 18 sessions. They reported 14 major defects and 5 minor defects.

scripted vs exploratory testing 2

In phase 2, the scripted team reported 2 major and 1 minor defect that the exploratory team didn’t find, but the exploratory team reported 3 major and 1 minor defect that the scripted team didn’t report.

This doesn’t take into account the relative complexities of the workflows that may have been chosen by the testers within these sessions and the test cases, but we can still draw some interesting conclusions.

What does it mean?

It would appear that an exploratory approach, and the responsibility and flexibility it engenders, results in a more effective form of testing. It may be possible to cover more ground by developing and adapting your test charters as the test sessions progress, based on what makes sense in context. This freedom is lacking in script-based testing and it can prevent defect discovery.

Sticking rigidly to scripts creates well-worn paths, and it’s only by deviating from those paths that we’re going to uncover all the defects. As mentioned several times by thought-leaders within the testing community, “If you imagine a product as a field of landmines and each landmine is a defect, then it’s pretty clear that treading the same path over and over is not the way to find them all.”

In the end, neither approach was perfect, because each team reported defects that the other team did not identify, even if the exploratory team did report more, overall.

Realistically, this may mean that the right approach, with regard to coming as close as possible to “minimal” defects, is going to be a mixture of the two. But, there are many benefits with the context-driven approach that speak in its favour. It requires less preparation time, less documentation, identifies issues earlier, and challenges testers to use analytical skills and deductive reasoning. They gain a deeper and more thorough understanding of the product and really act as advocates for the end user.

Conclusion

The end result demonstrates that exploratory testing does lead to reporting of more defects before go-live, resulting in a better product delivered by the team, and ultimately, more satisfied/fulfilled testers which are all desirable outcomes, any way you look at it.

About the Author

Mush Honda is QA Director at KMS Technology, a provider of IT services across the software development lifecycle with offices in Atlanta, GA and Ho Chi Minh City, Vietnam. He was previously a tester at Ernst & Young, Nexidia, Colibrium Partners and Connecture. KMS services include application management, testing, support, professional services and staff augmentation.

Do you agree? Feel free to post your comments, questions below.

 




Recommended reading

10 comments ↓

#1 Yevgeny

It’s like Villarriba and Villabajo!

#2 Piyush uphal

We prefer a mixed approach to avoid drawbacks of each method. Your study also indicates the same. isn’t it?

#3 Dayl

I myself like unscripted testing so I can just explore, however. It is absolutely paramount that very detailed notes be kept. A tester has got to be able to repeat what he/she did during the test event to be able to recreate a bug. Short of that I see no more serious benefit between the two.

#4 Sam

Exploratory testing makes a better process for small projects or part of the project is finished. Test script way of testing works only that there needed a full documentation. So this is only possible in the way that having full knowledge of project.

#5 shawn

shouldn’t automation handle the “scripted” part of testing and freeing up the manual tester to do more exploratory testing? I would prefer not to waste resources (people and time) that automation can do.

#6 Michael Fritzius

Scripted testing plus exploratory is an incredibly powerful combination.

I see exploratory testing being the engine behind writing great automated tests–high value bugs can be found, and then just enough automation can be written to catch those bugs later.

#7 Gaurav Khurana

Exploratory is always better , but yes as @shawn said some things which are very simple and cant be done
manually so easy should be left on script.

Yes but script should be perfect to cover all different paths as you have rightly gave the example of landmines.

So if we are relying on script our scripts should cover all possible paths to uncover all landmines.

In the last you have mentioned ‘context driven’ <– it comes under which approach as we were discussing scripted and exploratory here

#8 Gunasekaran Veerapillai

You can not choose between Scripted testing and Exploratory testing as an approach. They can be complemented. But in terms of the testers profile who are performing these two types of testing, there is great difference. Any tester can test the applications in a scripted approach but all of them can not do exploratory testing

#9 Puneet Sharma

That’s pretty interesting. We at Bugclipper don’t only rely on scripted testing, rather make sure it goes thru exploratory test cycle. I’d agree, combining both gives you best.

#10 Akshay

Found very interesting article. I would say Exploratory testing is well suited when requirements are changing frequently and scripted testing is well suited when requirement are stable. But I would suggest we need to adopt both the approaches because even though they are different but there objective is same (to find out more and more defects)…. Thanks again for sharing this article.

Leave a Comment