Learn What Is The Difference Between Test Plan, Test Strategy, Test Case, Test Script, Test Scenario And Test Condition With Examples:
Software Testing includes several basic as well as important concepts that every software tester should be aware of.
This article will explain the various concepts in Software Testing along with their comparison.
Test Plan vs Test Strategy, Test Case vs Test Script, Test Scenario vs Test Condition and Test Procedure vs Test Suite are explained in detail for your easy understanding.
=> Click Here For Complete Test Plan Tutorial Series
The above question asked by Sasi C. is the most often asked question in our Software Testing class and I always tell our participants that with the experience we hardly notice these words and that they become a part of our vocabulary.
But often, confusion surrounds these and in this article, I am trying to define few commonly used terms.
Various Software Testing Concepts
Enlisted below are the various Software Testing Concepts along with their comparison.
Let’s Start!!
What You Will Learn:
Difference Between Test Plan And Test Strategy
Test Strategy and Test plan are two important documents in the testing life cycle of any project. Here we are trying to give you an in-depth knowledge of test strategy and test plan documents.
Test Plan
A Test Plan can be defined as a document that defines the scope, objective, and approach to test the software application. The Test Plan is a term and a deliverable.
The Test Plan is a document that lists all the activities in a QA project, schedules them, defines the scope of the project, roles & responsibilities, risks, entry & exit criteria, test objective, and anything else that you can think of.
The Test Plan is as I like to call a ‘super document’ that lists everything there is to know and need. Please check this link for more information and a sample.
The Test Plan will be designed based on the requirements. While assigning work to the test engineers, due to some reasons one of the testers gets replaced by another one. Here, the Test Plan gets updated.
The Test strategy outlines the testing approach and everything else that surrounds it. It is different from the Test Plan, in the sense that a Test strategy is only a subset of the test plan. It is a hardcore test document that is to an extent generic and static. There is also an argument about at what levels test strategy or plan is used- but I really do not see any discerning difference.
Example: The Test Plan gives information about who is going to test at what time. For Example, Module 1 is going to be tested by “X tester”. If tester Y replaces X for some reason, the test plan has to be updated.
Test Plan Document
Test Plan is a document that provides complete information about testing tasks related to a Software Project. It provides details like Scope of the testing, Types of testing, Objectives, Test Methodology, Testing Effort, Risks & Contingencies, Release Criteria, Test Deliverables, etc. It keeps track of possible tests that will be run on the system after coding.
The test plan is obviously set to change. Initially, a draft test plan will be developed based on project clarity at that time. This initial plan will get modified as the project progresses. Test team Manager or Test Lead can prepare the test plan document. It describes the Specifications and is subject to change based on the same.
What to test, when to test, who will test, and how to test will be defined in the test plan. Test Plan will sort out a list of issues, dependencies, and the underlying risks.
Types Of Test Plan
Test Plans can be of different types based on the stage of testing. Initially, there will be a master test plan for the entire project execution. Separate test plans can be created for specific testing types like system testing, system integration testing, user acceptance testing, etc.
Another approach is to have separate test plans for functional and non-functional testing. In this approach performance, testing will have a separate test plan.
Contents of Test plan Document (IEEE-829 test plan structure)
It is difficult to draw a clear format for the test plan. The test plan format may vary depending on the project in hand. IEEE has defined a standard for test plans which are described as the IEEE-829 test plan structure.
Please find below IEEE recommendations for a standard test plan content:
- Test Plan Identifier
- Introduction
- Test Items
- Software Risk Issues
- Features to be tested
- Features not to be tested
- Approach
- Item Pass/Fail Criteria (or) Acceptance Criteria
- Suspension Criteria and Resumption Requirements
- Test Deliverables
- Test Tasks
- Environmental Requirements
- Staffing and Training needs
- Responsibilities
- Schedule
- Approvals
Suggested Read => Test Plan Tutorial – A Perfect Guide
Test Strategy
Test Strategy is a set of guidelines that explain the test design and determine how testing needs to be done.
Example: A Test Strategy includes details like “Individual modules are to be tested by the test team members”. In this case, who tests it does not matter – so it’s generic and the change in the team member does not have to be updated, keeping it static.
Test Strategy Document
The purpose of the test strategy is to define the testing approach, the types of tests, test environments, and tools to be used for testing and the high-level details of how the test strategy will be aligned with other processes. The test strategy document is intended to be a living document and will be updated** when we get more clarity on Requirements, SLA parameters, Test environment and Build management approach, etc.
Test strategy is intended for the complete project team that comprises of Project Sponsors, Business SMEs, Application/ Integration Development, System Integration partners, Data Conversion Teams, Build/Release Management Teams such as technical leads, architecture leads, and deployment and infrastructure teams.
** Some argue that test strategy once defined should never be updated. In most testing projects usually, it gets updated as the project progress.
Below are the important sections that a test strategy document should have:
#1) Project Overview
This section can begin by giving an overview of the organization followed by a brief description of the project in hand. It can include below details
- What was the need for the project?
- What objectives the project will attain?
Table of Acronyms: It is better to include a table with acronyms that the document reader might come up with while referring to the document.
#2) Requirements Scope
Requirement scope can include Application Scope and Functional scope
Application Scope defines the system under test and the impact on the system due to new or changed functionality. Related systems can also be defined.
| System | Impact (New or Changed functionality) | Related System |
|---|---|---|
| System A | New enhancements and bug fixes | • System B • System C |
Functional Scope defines the impact on different modules within the system. Here each related system with respect to functionality will be explained.
| System | Module | Functionality | Related System |
|---|---|---|---|
| System C | Module 1 | Functionality 1 | System B |
| Functionality 2 | System C |
#3) High-Level Test Plan
Test Plan is a separate document. In the test strategy, a high-level test plan can be included. A high-level test plan can include test objectives and test scope. Test scope should define both in scope and out of scope activities.
#4) Test Approach
This section describes the testing approach that will be followed during the testing life cycle.
As per the above diagram testing will be conducted in two phases i.e. Test Strategy & Planning and Test Execution. Test Strategy & Planning phase will be one time for an overall program whereas Test execution phases will be repeated for each Cycle of the overall program. The above diagram shows different stages and deliverables (outcome) in each phase of the execution approach.
Test approach should include below sub-sections
a) Test Schedule: Explain the proposed project timeline in this subsection
b) Functional Testing Approach: Using this subsection provides an overview of each phase and the respective entry and exit criteria. Different testing phases are Unit testing, System testing, System Integration testing, User Acceptance Testing, and End-to-End testing.
c) Testing Key Performance Indicators :
- Test case prioritization: Define the test case prioritization approach so that in case of time constraints, high priority scenarios can be executed by the test team. There should be an agreement between project stakeholders regarding the possible risks involved in not executing all the planned scenarios.
- Defect prioritization: Defect prioritization strategy is the next topic to cover here. Define priority level and give the description to each level like critical, high, medium, etc. Also
- Defect Turnaround Time: Defect turnaround time is defined as the time between when the defect was first raised and when the defect is fixed and comes for a retest. Quick turnaround ensures speedy testing and adherence to the project timeline. For each defect priority level, define the turnaround time.
| Priority Level | Defect Turnaround Time |
|---|---|
| 1 - Critical | Response Time: 2 hours or less Fix Ready for Migration: 1 business days or less |
#5) Test Coverage
This section describes the processes that the QA team will follow in order to optimize the coverage of business/functional requirements in test scenarios and test cases. Requirement Traceability Matrix: (RTM) can be used to trace all the requirements with respective test scenarios and test cases.
#6) Test Environment
Define the different QA environments available. Mention what testing will be done in which environment and by whom. Create an environment backup plan to take care of emergencies. Access to each environment should be regulated and called out with clarity.
Testing tools that are going to be used also can be mentioned in this section.
| Activity | Tool | Remarks |
|---|---|---|
| Test management | HP ALM | Mention the reason for using this tool |
| Defect management | JIRA | Mention the reason for using this tool |
#7) QA Deliverables and Metrics
List out all the QA deliverables
| S. No. | Deliverable |
|---|---|
| 1 | Test Strategy Document |
| 2 | Requirement Traceability Matrix |
| 3 | ST Test Scripts |
| 4 | Test Summary Report |
| 5 | Automation eligible scenario list |
List out all the QA metrics
| # | Metric Name | Metric Definition | Metric Formula | Metric unit of measure | Reports in which the metrics to be used |
|---|---|---|---|---|---|
| 1 | Requirements Coverage Metrics(RCM) | The coverage of requirements by QA | Ratio of # of requirements tested to # of requirements identified | % | Weekly QA status report, Test summary report |
| 2 | Test Coverage | The coverage of test case executed | Ratio of number of test cases executed /number of test cases planned | % | Daily Execution report, Weekly QA status report, Test summary report |
#8) Defect Management
Clearly define a defect management strategy by creating a defect workflow, defect tracking methodology & defect triage process. Mention defect responsibility for each tester’s roles. Periodic defect analysis and root cause analysis will improve the overall quality of testing
#9) Communication Management
Set guidelines for status reports, status meetings, and onsite-offshore communication.
#10) Assumptions, Risks, and Dependencies
Describe assumptions on which the project is based. These may include timing, resources, and system capabilities. Describe any dependencies such as other projects, availability of temporary resources, other deadlines that may impact the project
#11) Appendix
Include stuff like Roles & Responsibilities, Work Time Zone, and References in this section
Further reading => Guide to Writing a Good Test Strategy document.
Test Plan Vs Test Strategy
| TEST PLAN | TEST STRATEGY |
|---|---|
| It is derived from software requirement specification(SRS). | It is derived from the Business Requirement document(BRS). |
| It is prepared by the test lead or manager. | It is developed by the project manager or the Business analyst. |
| Test plan id, features to be tested, test techniques, testing tasks, features pass or fail criteria, test deliverables, responsibilities, and schedule, etc. are the components of the test plan. | Objectives and scope, documentation formats, test processes, team reporting structure, client communication strategy, etc. are the components of test strategy. |
| If there is a new feature or a change in the requirement that is happened then the test plan document gets updated. | Test strategy maintains the standards while preparing the document. It is also called as Static document. |
| We can prepare the test plan individually. | In smaller projects, test strategy is often found as a section of a test plan. |
| We can prepare a Test plan at the project level. | We can use Test strategy at multiple projects. |
| It describes how to test , when to test, who will test and what to test. | It describes what type of technique to follow and which module to test. |
| We can describe about the specifications by using a Test Plan. | Test strategy describes about the general approaches. |
| Test Plan will change over the course of the project. | Test Strategy usually will not change once approved. |
| Test plan is written after requirement sign off. | Test strategy is made before the test plan. |
| Test plans can be of different types. There will be a master test plan and separate test plan for different types of testing like system test plan, performance test plan, etc. | There will be only one test strategy document for a project. |
| Test plan should be clear and concise. | Test strategy provides overall guidance for the project in hand. |
The difference between these two documents is subtle. A test strategy is a high-level static document about the project. On the other hand, the test plan will specify what to test, when to test, and how to test.
Difference Between Test Case And Test Script
In my opinion, these two terms can be used interchangeably. Yes, I am saying there is no difference. The test case is a sequence of steps that help us perform a certain test on the application. The test script is also the same thing.
Now, there is one school of thought that a test case is a term used in the manual testing environment and test script is used in an automation environment. This is partly true, because of the comfort level of the testers in the respective fields and also on how the tools refer to the tests (some call test scripts and some call them to test cases).
So in effect, test script and test case both are steps to be performed on an application to validate its functionality whether manually or through automation.
Further reading => How to Write Effective Test Cases? and Test Case Example Template.
| TEST CASE | TEST SCRIPT |
|---|---|
| It is a step by step by procedure that is used to test an application | It is a set of instructions to test an application automatically. |
| The term Test Case is used in the manual testing environment. | The term Test Script is used in automation testing environment. |
| It is done manually. | It is done by scripting format. |
| It is developed in the form of templates. | It is developed in the form of scripting. |
| Test case template includes Test Suit ID, Test Data, Test procedure, Actual results, Expected results etc. | In Test Scrip,t we can use different commands to develop script. |
| Is used to test an application. | It is also used to test an application. |
| It is the base form to test an application in sequence. | Once we develop, the script will run it multiple times until the requirement is changed. |
| Example: We need to verify the login button in an application, The steps include: a) Launch the application. b) Verify if the login button is displaying or not. | Example: We want to click an image button in an application. The script includes: a) Click the Image Button. |
Difference Between Test Scenario And Test Condition
Test Scenario: It is a way to define all the possible ways to test an application. It is a single statement to cover all possible ways to test an application.
Test Condition: Test Condition is the specification that a tester must follow for testing an Application.
This is a one-line pointer that testers create as an initial, transitional step into the test design phase. This is mostly a one-line definition of “What” we are going to test with respect to a certain feature. Usually, test scenarios are input for the creation of test cases.
In agile projects, Test scenarios are the only test design outputs and no test cases are written following these. A Test scenario might result in multiple tests.
Examples of Test Scenarios:
- Validate if a new country can be added by the Admin
- Validate if an existing country can be deleted by the Admin
- Validate if an existing country can be updated
Test conditions, on the other hand, are more specific. It can be roughly defined as the aim/goal of a certain test.
Example Test Condition: In the above example, if we were to test the scenario 1, we can test the following conditions:
- Enter the country name as “India”(valid )and check for the addition of the country
- Enter a blank and check if the country gets added.
- In each case, the specific data is described and the goal of the test is much more precise.
Further reading => 180+ Sample Test Scenarios for Testing Web and Desktop Application.
| TEST SCENARIO | TEST CONDITION |
|---|---|
| It is a process to test an application with all possible ways. | Test conditions are the static rules should be followed to test an application. |
| Test scenarios are an input for the creation of test cases. | It gives the main goal to test an application. |
| Test scenario covers all possible cases to test an application. | Test condition is very specific. |
| It reduces the complexity. | It makes a system bug free. |
| Test scenario can be a single or a group of test cases. | It is the goal of test cases. |
| By writing scenarios it will be easy to understand the functionality of an application. | Test condition is very specific. |
| These are one line statements to explain what we are going to test. | Test Condition describes the main goal to test an application. |
| Examples test scenarios: #1) Validate if a new country can be added by the Admin. #2) Validate if an existing country can be deleted by the admin. #3) Validate if an existing Country can be updated. | Examples test Conditions: #1) Enter the country name as “India” and check for the addition of the country. #2) Leave blank fields and check if the country gets added. |
Difference Between Test Procedure And Test Suite
The test procedure is a combination of test cases based on a certain logical reason, like executing an end-to-end situation or something to that effect. The order in which the test cases are to be run is fixed.
Test Procedure: It is nothing but the Test Life Cycle. There are 10 steps in the Testing Life Cycle.
They are:
- Effort Estimation
- Project Initiation
- System Study
- Test plan
- Design Test Case
- Test Automation
- Execute Test Cases
- Report Defects
- Regression Testing
- Analysis and Summary Report
For Example, if I was to test the sending of an email from Gmail.com, the order of test cases that I would combine to form a test procedure would be:
- The test to check the login
- The test to compose an email
- The test to attach one/more attachments
- Formatting the email in the required way by using various options
- Adding contacts or email addresses to the To, BCC, CC fields
- Sending an email and making sure it is showing in the “Sent Mail” section
All the test cases above are grouped to achieve a certain target at the end of them. Also, test procedures have a few test cases combined at any point in time.
The Test suite, on the other hand, is the list of all the test cases that have to be executed as a part of a test cycle or a regression phase, etc. There is no logical grouping based on functionality. The order in which the constituent test cases get executed may or may not be important.
Test Suite: The Test Suite is a container that has a set of tests which help the testers in executing and reporting the test execution status. It can take any of the three states i.e. Active, in progress and completed.
Example of the Test Suite: If an application’s current version is 2.0. The previous version 1.0 might have had 1000 test cases to test it entirely. For version 2 there are 500 test cases to just test the new functionality that is added in the new version.
So, the current test suite would be 1000+500 test cases that include both regression and the new functionality. The suite is a combination too, but we are not trying to achieve a target function.
Test suites can contain 100s or even 1000s of test cases.
| TEST PROCEDURE | TEST SUITE |
|---|---|
| It is a combination of test cases to test an application. | It is a group of test cases to test an application. |
| It is a logical grouping based on the functionality. | There is no logical grouping based on the functionality. |
| Test Procedures are deliverable products in the software development process. | It is executed as a part of the test cycle or regression. |
| The order of execution is fixed. | The order of execution may not be important. |
| Test procedure contains end to end test cases. | Test suite contains all new features and regression test cases. |
| Test procedures are coded in a new language called TPL(Test Procedure language). | Test suite contains manual test cases or automation scripts. |
| Creation of Test Procedures is based on the end to end test flow. | Test suites are created based on the cycle or based on the scope. |
Conclusion
Software Testing Concepts play a major role in the Software Testing Life Cycle.
A clear understanding of the above-discussed concepts along with their comparison is very important for every Software Tester to carry out the testing process effectively.
Usually, articles like these are excellent starting points for deeper discussions. So, please contribute your thoughts, agreements, disagreements and anything else, in the comments below. We look forward to your feedback.
We also welcome your questions about software testing in general or anything related to your testing career. We will address these in more detail in our upcoming posts in the same series.
Happy Reading!!
=> Visit Here For Complete Test Plan Tutorial Series
