What is Exploratory Testing?
“Exploratory testing” – as the name suggests, is a simultaneous learning, test design, and test execution process. We can say that in this testing test planning, analysis, design and test execution, are all done together and instantly.
This testing is about exploring the system and encouraging real-time and practical thinking of a tester.
In this series we have covered following Tutorials:
Tutorial #1: What is Exploratory Testing in Software Testing (This Tutorial)
Tutorial #2: Using Tours to Ensure Complete Exploratory Testing
Tutorial #3: Exploratory Testing vs Scripted Testing
Tutorial #4: Exploratory Testing with HP Sprinter
Tutorial #5: Top 17 Exploratory Testing Tools
What You Will Learn:
- Exploratory Testing Examples
- Testing Approach
- Session-based Exploratory Testing
- Pair Based Exploratory Testing
- Exploratory Testing Techniques
- Difference between Exploratory Testing and Ad-hoc Testing
- Exploratory Automated Testing (EAT)
- Types of Exploratory Testing
- Agile Exploratory Testing
- How to Think Beyond Traditional Testing Boundaries in Exploratory Testing
- How to look at a product from different perspectives?
- Recommended Reading
In layman terms, exploratory testing involves concurrent test case design and test execution of an application or system under test. The tester will create or write down a test idea to give direction, and explore the system while testing to further create critical, practical and useful tests for the successful testing of an application.
This requires minimal planning. Testers continuously make a decision on her next step of action. It completely depends upon the tester's thought process.
Sometimes this testing can be more beneficial than the formal testing approach for finding some subtle defects which go missing in formal testing.
Consciously or unconsciously each and every tester would have done exploratory testing at some point in their career.
As we all know, a learner will learn better through hands-on experience rather than cramming the theory.
Same way, a tester will know the application better only while exploring and learning about all the functionality it provides by itself. It is always good to have a customer and business perspective while testing to ensure successful testing of an application.
For Example, if you open a shopping website, you have a general idea that this shopping website will let you shop by selecting a product of your choice and then paying for the same.
During this process, you might learn that the website provides you with virtual human look-alike which helps you in product selection process. You also found that you can order a number of products for home trial or that you can make payment through rewards points of some banks, etc.
As a tester, you not only need to verify whether a system is working as expected but also check if that system is not behaving in a way which is not expected.
Few things to remember while performing this testing:
- Your mission should be clear.
- Make sure to create notes and report on what you are doing and how a system is behaving, which could be a potential bug.
- Learn, observe and then come up with new test cases.
Exploratory Testing Examples
A home care service provider website with the following components:
- Order History
- Technician allotment
A general idea to start exploratory testing will be to Login or book a service.
How to Cover Test Cases?
In the above Example, the idea is to start with functionality based on your knowledge. As you learn and observe more about the application, you can govern your next set of test cases.
I was once included in a small project which involved the addition of a new Mutual fund in the application. My task was to test the application to make sure that the new Mutual fund is available for users to buy and check if the associated valuation is correct. I had only 2 days to complete my testing.
Being given the tight deadline and severity of testing, I used the exploratory approach of testing. My goal was to test new features and to find violations of compatibility requirements.
The above-mentioned goal became my charter for this test session.
Following test cases were evolved during this testing:
- Testing to make sure that the new Mutual fund has been added to the application.
- New MF is bought successfully.
- Valuation of new MF is correct.
- Tried to buy new MF for an existing portfolio.
- Can new MF be added to all Portfolios?
- Impact of New MF on a valuation of existing.
- So in other test cases were evolved.
I prepared notes and reports during my testing to discuss my observation with the BA and client involved.
The fundamental strategy of exploratory testing is to have a plan of attack. Start testing with your idea and improvise new test cases based on your knowledge and observation.
Exploratory Testing of IRCTC Website
- Make use of heuristics to guide testing.
- Test cases execution and test case creation go hand in hand.
- Test cases keep on evolving based on tester observation and learning.
- Different testing techniques like Boundary value analysis, equivalence testing, etc. can be applied to ET.
- Session-based ET can be used to make it more structured and focused.
- Testers can branch out there ideas but never stray from your mission.
- ET testing does not use scripts, instead depends on tester's intuition, skill, and experience.
Benefits of this testing include:
- Promote real-time thinking and helps in uncovering more defects.
- Promote use cases and scenario-based testing.
- Minimal documentation, maximum testing.
- Emphasis is more on learning and broadening the horizon of a tester.
- Avoid duplicate work.
- Useful when you want to audit other tester’s work.
Demerits are enlisted below:
- Testing depends on tester experience, skill, and knowledge.
- Require time to learn the application. Tester is more likely to miss if they know less about the application.
- Not appropriate for projects with long execution time.
Session-based Exploratory Testing
While doing exploratory testing, it’s very difficult for testers to put into words on how much he has tested and on what basis.
Basically, it’s difficult to quantify the work and the time spent. However, in every project, we need to provide metrics, estimates and progress report to team leads and the managers. As the saying goes, “if you can’t quantify it, you cannot manage it”.
Session-based testing is a time basis approach to perform this testing which helps in managing and tracking. It includes a dedicated time-boxed testing session with no interruption from email, phone, messages etc.
Testing tasks are divided into sessions.
Following are the components of session-based testing (SBT):
- Mission: Mission shout out the purpose of the session and in a way provide the focus for the tester. It will also include a session time duration.
- Charter: Includes the scope of the testing. Basically, an agenda detailing the goals that need to be completed during the session.
Example of Test Charter for Login Functionality of Home care service website:
- Session: Pre-defined time-boxed testing session without any interruption. Each session can have the following duration:
- “Short” (60min)
- “Normal” (90min)
- “Long” (120 min)
- Session report: Include notes and lightweight report to provide metrics to the leaders and managers. It gives details about the charter session remaining or done, session setup time, scenario tested, about the testing process, a list of bugs and the issues found and other information for the metrics.
- Session De-brief: A short meeting or stand-up between the tester and Test Lead/Manager to review the test session findings.
Managers can get hands-on following metrics based on session report:
- The number of sessions completed and remaining.
- The number of bugs reported.
- Time spent on session setup.
- Time spent on testing.
- Time spent on analyzing issues or problems.
- Features covered.
To summarize the above:
SBT allows for accountability is exploratory testing and offers better management of time spent on testing. It also increases productivity and provides a better grasp on bug detection. It is a great way to provide team leads and managers with metrics to check the project progress.
Pair Based Exploratory Testing
Pair Testing is an approach in which two people test the same thing/feature of the application at the same time by sharing a PC. They continuously share their thoughts and ideas. During this testing, one person takes charge of the keyboard whereas the other person suggests test cases and takes note.
It is always helpful to have a good communication between the partners so that both are aware of what is being done and why. A pair in which strength of the testers mutually complements their weakness is considered as a strong grouping.
Such pairing benefits both the parties and each can learn something from their partner. It is also a good way to train new resources by pairing them with experienced resources.
Benefits of Pair Testing
- Helps a tester to focus on the task at hand.
- Encourage mutual trust and respect among partners.
- Brainstorming between paired testers usually, lead to more constructive ideas.
- Avoid tunnel vision.
- There is a lesser chance of others interrupting them.
Exploratory Testing Techniques
Tours: It is a simple technique which allows a tester to use his imagination and think of himself as a tourist exploring a city he visits. Here an application to test is the city and the testers are the tourists. It is very difficult to explore the entire city unless you have a lot of time and money in your hand, so a tourist needs to have a plan with a certain goal in mind.
A tourist can take the following tours:
- Guidebook tour – Testing highlighted feature of the application. Use user-based scenarios.
- Exploring the history of the city – Test old features of an application.
- Money tour, which means making sure that all the critical features in reference to the customer or client are tested and working successfully.
- Crime spree tour – Enter invalid input and test negative scenarios.
- Back alley tour – Test the least used features of the application.
- Boring tour – Spend minimum time on each screen of the application, fill minimum fields and take the shortest path. This will help with the default value and validation testing.
While taking a tour, you always have the choice of taking any route. You can navigate through software and find a unique path to test the feature.
Below are some tips/tricks that you can use in ET:
- Divide the application into modules and bifurcate modules into different pages. Start your ET from the pages. This will give the right coverage.
- Make a checklist of all the features and put a check mark when that is covered.
- Start with a basic scenario and then gradually enhance it to add more features to test it.
- Test all the input fields.
- Test for the error message
- Test all the negative scenarios.
- Check the GUI against the standards.
- Check the integration of the application with other external applications.
- Check for complex business logic.
- Try to do the ethical hacking of the application.
Factors which affect ET are as follows:
- The objective of the project
- Testing strategy
- The testing goal of a particular phase
- Available tools and facilities
- Testers role and skills
- Available time
- Management support
- Peer support
- Available resources (study materials, test conditions etc)
- Clients interest
- Understandability of the product.
- The UI of the application
- The functionality of the application
- Previous test results
- Risks associated with the application
- Previous defects
- Recent changes
- Types of data to use for testing
- Type of user who will be using it
Instead of asking the testers what to run, we leave it to tester’s judgment to decide what they want to test and how they want to test.
Difference between Exploratory Testing and Ad-hoc Testing
Do not confuse ET with Ad-hoc testing.
- Ad-hoc testing refers to a process of unscripted, unplanned and impromptu defect searching whereas exploratory testing is a thoughtful methodology to Ad-hoc testing.
- Ad-hoc testing is a hit and trial method of finding a bug whereas ET is not. In ET approach, a tester learns about the system as they explore and eventually evolve the tests using the acquired knowledge.
- Ad-hoc testing is an unstructured activity whereas ET is somewhat a structured activity.
Exploratory Automated Testing (EAT)
Exploratory Automated Testing is a method that helps a tester in streamlining bug reporting & reproduction, snapshots gathering and in preparation of future regression suit. It's a process that combines automation testing with Exploratory testing.
There are two types of EAT approach:
- Passive EAT
- Active EAT
Passive EAT can be performed by a single tester or in a pair as well. In this methodology, usually, a tool, which captures and records every single activity performed by a testing resource(s) and is installed on the resource’s PC.
Passive EAT is similar to the ET that is performed manually as there is no change in the way the tests are executed apart from crafting the test result based on the captured session. These test results can be used for reporting and reenacting of recorded actions later in time.
The installed video tool helps a tester with test case recording and defect reporting.
It also has few other benefits like:
- Provides clear steps to reproduce the bugs.
- Reproducing defects is easier even when the defect reporter is not available.
- Do away with the conflicts between the testing and development team when an intermittent bug is reported.
- Helps in performance testing by getting the system response time at the specific point in time.
Here are few other points to be taken into consideration before Passive EAT:
- It is advised to perform a pilot test before completely adapting the tool for Automated EAT. This is to ensure that the time required for re-designing of the test logs created during the test session is not more than test execution. If so, then the team needs to take a mutual decision on the following:
- If at all test automation is required for the particular project.
- If the tool being used needs to be changed.
- If the performance of the tool being used can be optimized.
- The tool used for performing automated EAT needs to be installed on every testing resource involved in testing. It is also a good idea to involve the developers which can be achieved by either giving the developers VPN or remote access to test machines or by installing the tool in the development environment.
- It is always a good idea to have GUI object of the application organized in the test tool so that when the time comes for analyzing the bug or an issue, the object is recognizable due to a meaningful name.
- It is a great practice to give a meaningful name to the GUI object used in AUT and keep them organized for later use.
Now, let's move on to the second approach.
It is advisable to perform Active EAT with Pair Testing. In this approach, the Keyword Driven testing is used in sync with Session testing. One tester creates the automated test script and the second tester executes the test scripts created by the first tester.
Creation of automation test scripts in this approach takes a different path than in conventional testing. Automated test scripts are made during testing and what has been discovered in the previous tests determine their design.
A closure phase is executed at the end of the testing session. And it should have the following tasks:
- Testers involved should swap roles so that the testing resource who created the test script has a chance to re-execute the scripts to confirm the reliability & robustness of the created suite.
- A brief description along with few identifying characteristics should be provided for every automated test script.
- A criterion needs to be defined to identify which Automated test scripts can be used for Regression test.
Benefits of EAT
- At the start of each session, already created automated test scripts are executed thus enhancing the test coverage every-time.
- Better bug reporting and documentation for defect reproduction.
- EAT provides enough evidence and documentation for a stakeholder to see the progress.
Types of Exploratory Testing
Given below are a few types of ET:
1) Freestyle ET:
Exploration of application in ad-hoc style.
In this type of ET, there are no rules, no account for coverage, etc. However, this type of testing is good when you need to familiarize with the application quickly, when you want to verify the work of the other testers, and when you want to investigate a defect or want to do a quick smoke test.
2) Scenario-based ET:
As the name itself suggests, testing done is scenario-based. It starts with real user’s scenarios, end-to-end scenarios or test scenarios. After initial testing, testers can inject variation as per their learning and observation.
Scenarios are like a generic guide for what to do during ET. Testers are encouraged to explore multiple possible paths while executing a scenario to ensure all possible path to feature work. Testers should also ensure to gather as many scenarios as possible from different categories.
3) Strategy based ET:
Known testing techniques such as Boundary value analysis, equivalence technique, and risk-based technique that are combined with exploratory testing. An experienced tester or a tester who is familiar with the application is appointed for this type of testing.
Agile Exploratory Testing
Even if you have not worked in an agile environment, I am sure you must have read or heard about it because of its growing popularity. Agile methodology has short sprints and tight deadlines which gives a team couple of weeks to finish planning, estimation, development, coding, testing, and release.
Exploratory testing becomes handy in such tight deadlines because in this testing approach emphasis is on the quick and useful result. Once you have understood the requirement, you can start testing based on your experience and knowledge.
Once you are familiarized with the application features and behavior, you can design more test cases to validate the application functionality and detect unplanned bugs. As it’s a freestyle testing approach, you do need to document everything. However, you need to maintain notes and a brief report on what you have tested, bugs and issues found etc.
Merits of Exploratory in Agile
- Proving feedback to the developers as soon as possible.
- A broader variety of defects are uncovered.
- A diverse group of a resource such as a developer, tester, BA, designers can perform ET as there are no scripted test cases and each brings a different perspective.
- Scouting done in ET helps in exploring new territories and exposing critical bugs.
- In case of Iterative coding of an application, ET can focus on testing new features while automation does regression and backward compatibility testing.
- In case of unstable requirement, ET can help in testing new requirement within a limited time.
Points to Remember:
1. Requires different skills: The testers performing ET need to have good listening, reading, thinking and reporting skills. Domain experience is required as there are no scripts and test cases.
2. Sometimes it's difficult to report a bug: While in an ET flow, we may encounter a defect but we may not be able to reproduce it. This is because we are not tracking the testing steps and we may forget the exact steps to reproduce that issue.
3. Can be done as recreation activity: I personally do ET when I want a break from my regular test execution cycle. But many teams have ET as a separate phase of their testing cycle.
4. It can be done for all testing phases: We can implement ET before the beginning of any testing phase. You can perform ET even before functional testing phase.
5. Rapid feedback: ET requires rapid feedback on the issues and any anomalies encountered.
6. Critical Thinking and diverse ideas: This testing requires critical thinking. Testers should be able to reproduce, review and express their ideas in a logical way. A tester can implement her experience in the various technologies and domains they worked on.
How to Think Beyond Traditional Testing Boundaries in Exploratory Testing
“I really appreciate your concern for the product and making us helpful in an understanding end-user perspective. It’s going to be very helpful. Thanks for the good work and keep it up!!!”
This was the last e-mail of an email chain with 21 emails from our client. It was midnight, and our product release was delayed due to a critical bug we found. You might think, what is new in that? It may happen many times. But, this was really different as the critical bug we reported was not a result of any documented test case.
After completing regression testing for the last time that evening, I was just playing with the product. What does that mean? You are free to do what you are not supposed to do. Based on my experience and project knowledge, I had some ideas on how to test the product apart from our typical test repository, called Exploratory Testing.
The exploratory testing done found a critical bug related to a server hang issue while doing something unexpected.
Being a fan of exploratory testing, I love to explore the product in different ways. For me, the definition of software is:
“It should do what it is supposed to do, and it should not do what it is not supposed to do.”
Limiting testing boundaries to check whether products that are supposed to work are working makes you an incomplete tester. In fact, a tester’s life starts when documented regression testing ends and results are updated. Looking at products from different perspectives and understanding end-user requirements in different scenarios make a big difference. So today, let’s understand together, how that difference can be made:
How to look at a product from different perspectives?
#1. Understand customer/end-user
Software testing is all about checking the quality of the product in terms of customer satisfaction. How do you know a customer’s viewpoint? The answer is simple – you have to be the customer. OK, let me make a correction. Being customer will not be enough. You need to understand how the customer wants to handle the product. No two customers who bought same raw materials will prepare the same recipe. Yes, the product we develop/deliver is a raw material for customer’s businesses and they have a different mindset while using it.
As a software tester, we need to check the purpose of the product and not the object or aspect of it.
Let me give you some real-life practical examples:
- Scissors were never limited to cut paper only. Cutting is the purpose and not the paper (object).
- Cell phones were never limited to only calling, but “able to call” has always been the basic purpose.
- Storage boxes are used to store, but the safety of the material stored is as important as storage.
Understanding stakeholders and a wide range of their expectations should be the baseline of exploratory testing.
#2. A mindset
While looking for (let’s say) a job employment ad, do you see that jackpot and in between the pages with the bold font? Most of us do not (believe me, it’s true). Because we have instructed our mind to look for what is useful or to be checked. Anything else is of no use, so the mind denies us of recognizing it.
Open your mind, and do not set any expectations when you start exploring a product. Always remember, it’s not OK if the product is doing what it is supposed to do. It is also important that it should not do what it is not supposed to do.
I remember one classic example:
In Linux, ‘cat' command is used to check the content of a file and the ‘ls' command is to check the content of the directory. Working with Linux and being in software testing for five years, I never thought to do cat <dir name> because my mind was set; if I needed dir content, I need to use ‘ls'. That worked, but the reverse side of the expectation is that the product was not supposed to behave the way it was not supposed to, was wrong. One of our customers, who did not know Linux well, did cat <dir name> by mistake and the system crashed. We paid for this mindset.
Always be ready to make mistakes with the software because that is what end-user is going to do. To test the software, you have been trained, but the end-user will not be as trained as you or he/she will not be as much of a technical expert as you. Also, he/she will do anything with the software when they are in trouble.
Think about those scenarios, and provide testing feedback. Life of the software and yours (as a tester) will rock.
#3. Know the competitors
While testing any software application for your client, did you ever attempt to know and understand other software with the same purpose? Did you ever suggest some useful functionality you observed in a competitor’s product? It does not fall in our job description, is the typical answer. But do you know the benefit of doing it?
Here are few real-life examples to make you understand the point:
- Don’t you like the designer that not only stitches your dress but also provides you input about matching accessories most?
- Don’t you like the pizza brand who not only makes great pizzas but home delivers on time most?
- Don’t you like the photographer who not only takes good photographs but suggests a different kind of frames for the photoshoot?
Everyone wants to have something extra for what they pay for. Our analysis of competitive software can work the same way for us. The customer always likes to hear valuable suggestions – mainly comparative suggestions to make the product more useful or marketable.
Also, this kind of comparison and analysis of the same range of products makes our analysis more powerful and eventually we create a treasure to which we can go back at any moment and find something useful.
Exploratory doesn’t come under a conventional way of testing but still, it is a very powerful way of testing.
It brings out the out of box thinking of a tester and encourages them to come up with practical and real-time test cases for finding a defect. Its freestyle nature gives it an edge over the other testing types and can be performed anywhere, be it a project using Agile or waterfall or any other project which requires minimal documentation.
The success of exploratory testing depends on numerous intangibles like the skill of a tester, the ability to create effective test cases, their experience, and the knack to follow their gut feeling.
It’s imperative to remember that ET is an adaptive process rather than a predictive one and it’s essential to maintain a healthy balance between exploratory and scripted or regular testing.
Are you a tester who has typical Exploratory testing experiences? We are waiting to hear your thoughts. Feel free to share them in the comments section below.
Next Tutorial #2: How to Use Tours to Ensure Complete Exploratory Testing