What is the difference between Performance Test Plan and Test Strategy?
In this Performance Testing series, our previous tutorial, explained about Functional Testing Vs Performance Testing in detail.
=> Click Here For Complete Performance Testing Tutorials Series
In this tutorial, you will learn about the difference between Performance Test Plan and Test Strategy and the content to be included as part of these documents.
Let’s understand the difference between these two documents.
Table of Contents:
Performance Test Strategy
Performance Test Strategy document is a high-level document which gives us information on how to carry out performance testing during the testing phase. It tells us how to test a Business requirement and what approach is required to successfully deliver the product to the end client.
This will have all the information about the Business process at a very high level.
This document is usually written by Performance Test Managers based on their prior experience as there will be only limited information available as this document is prepared during the initial stages of the project i.e, during the Requirement Analysis phase or after the Requirement Analysis phase.
So, in other words, a Performance Test Strategy document is nothing but a direction that you set at the start of the project with the approach that you are going to take, in order to achieve the Performance testing goals.
A typical Performance Test Strategy document contains the overall goal of Performance testing as what will be tested? which environment will be used? which tools will be used? what types of testing’s will be carried out? Entry and Exit criteria, what Risks of a stakeholder are mitigated? and few more which we shall look in detail as we move further in this tutorial.
The above diagram explains that the Performance Test Strategy document is created during or after the Requirement analysis phase of the project.
Performance Test Plan
Performance Test Plan document is written at a later stage in the project when the requirements and design documents are almost frozen. The Performance Test Plan document has all the details of the schedule to implement the strategy or Approach which was described during the Requirement Analysis Phase.
As of now, the Design documents are almost ready, the Performance Test Plan contains all details about the scenarios to be tested. It also has more details about the Environments which are used for Performance Test Runs, How many cycles of Test runs, Resources, Entry-Exit criteria and more. The Performance Test Plan is either written by the Performance Manager or the Performance Test Lead.
The above diagram clearly explains that the Performance Test Plan is created during the project Design or after the Design Phase based on the availability of the Design documents.
Contents of Performance Test Strategy Document
Let’s now see what all should be included in a Performance Test Strategy document:
#1) Introduction: Give a brief overview of what a Performance Test Strategy document will contain for that particular project. Also, mention the teams which will use this document.
#2) Scope: Defining the scope is very important because it tells us what exactly will be the Performance Tested. We need to be very specific while defining the scope or any other section.
Never write anything generalized. Scope tells us what exactly will be tested for the entire project. We have In scope and Out of scope as a part of the scope, In scope describes all the features which will be Performance Tested and Out of scope describes the features which will not be tested.
#3) Test Approach: Here we need to mention about the approach that we are going to follow for our Performance Tests like each script will be executed with a single user to create a baseline and then this baseline tests will be used as a reference for Benchmarking at a later point of time during Test Runs.
Also, each component will be tested individually before integrating them together and so on.
#4) Test Types: Here we mention the different types of tests to be covered, like Load Test, Stress Test, Endurance Test, Volume Test etc.
#5) Test Deliverables: Mention what all deliverables will be provided as a part of Performance Testing for the Project like Test Run Report, Executive Summary Report etc.
#6) Environment: Here we need to mention the details of the environment. Environment details are very important as it describes what operating systems will be used for Performance Testing.
If the environment will be a replica of production or will it be sized up or sized down from production and also the ratio of sizing up and sizing down i.e will it be half the size of the production or will it be double the size of the production?
Also, we need to clearly mention any Patches or security updates to be considered as a part of the environment set up and also during the Performance Test Run.
#7) Tools: Here we need to mention all the Tools which will be used like Defect Tracking tools, Management tools, Performance Testing, and Monitoring Tools. Some Examples of tools for defect tracking is JIRA, For Management of documents like Confluence, for Performance Testing Jmeter and for monitoring Nagios.
#8) Resources: Details of the Resources required for the Performance Testing Team are documented in this section. For Example, Performance Manager, Performance Test Lead, Performance Testers etc.
#9) Entry & Exit Criteria: Entry and Exit criteria will be described in this section.
For Example,
Entry Criteria – Application should be functionally stable before deploying the build for Performance Testing.
Exit Criteria – All the major defects are closed and most of the SLAs are met.
#10) Risk and Mitigation: Any Risks which will affect the Performance Testing must be listed here along with the mitigation plan for the same. This will help any risks from occurring during Performance testing or at least a workaround for the Risk will be planned well in advance. This will help with completing the Performance Test Schedules on time without affecting the deliverables.
#11) Abbreviations: Used for Abbreviations. For Example, PT – Performance Test.
#12) Document History: This contains the document version.
Contents of Performance Test Plan Document
Let’s take a look at what all should be included in a Performance Test Plan document:
#1) Introduction: It is all the same as stated in the Performance Test Strategy document, rather we just mention Performance Test Plan instead of Performance Test Strategy.
#2) Objective: What is the objective of this performance testing, what is achieved by conducting performance testing i.e, what are the benefits of doing performance testing should be clearly mentioned here.
#3) Scope: Scope of Performance Testing, both in scope and out of scope business process is defined here.
#4) Approach: Overall approach is described here, how performance testing is carried out? What are the prerequisites for setting up the environment? etc are included.
#5) Architecture: Details of the Application Architecture should be mentioned here, like the total number of Application servers, Web servers, DB servers, Firewalls, 3rdd party application Load generator machines etc.
#6) Dependencies: All pre-performance testing actions should be mentioned here, like the components to be performance tested are functionally stable, environment is scaled to a production like one and is available or not, Test date is available or not, Performance Testing tools are available with licenses if any and so on.
#7) Environment: We need to mention all the details of the system like IP address, how many servers etc. We should also mention clearly as how the Environment should be set up like the prerequisites, any patches to be updated etc.
#8) Test Scenarios: The list of scenarios to be tested are mentioned in this section.
#9) Work Load Mix: The work Load mix plays a vital role in the successful execution of the performance test and if the workload mix doesn’t predict the real-time end-user action, then all the test results go vain and we end up with poor performance in production when the application goes live.
Hence it is necessary to properly design the workload. Understand how the users are accessing the application in production and if the application is already available or else try to get more details from the business team to properly understand the application usage and define the workload.
#10) Performance Execution cycles: Details of the number of performance test runs will be described in this section. For Example, Base Line test, Cycle 1 50 user test etc.
#11) Performance Test Metrics: The details of the metrics collected will be described here, these metrics should be in acceptance criteria with the agreed performance requirements.
#12) Test Deliverables: Mention the deliverables, and also incorporate the links to the documents where ever applicable.
#13) Defect Management: Here we need to mention how the defects are handled, the severity levels and priority levels should also be described.
#14) Risk Management: Mention the risks involved with the mitigation plan like if the application is not stable and if high priority functional defects are still open, will it affect the schedule of the performance test runs and as said earlier this will help any risks from occurring during Performance testing or at least a workaround for the Risk will be planned well in advance.
#15) Resources: Mention the team details along with their roles and responsibilities.
#16) Version history: Keeps a track of the document history.
#17) Document Reviews and Approvals: This has the list of people who will review and approve the final document.
Thus, basically Performance Test Strategy has an approach to Performance Testing and Performance Test Plan has the details of the approach, hence they go together. Some companies just have a Performance Test Plan which has Approach added to the document, whereas some have both strategy and plan document separately.
Tips to Develop These Documents
Follow the below guidelines while designing the strategy or a plan document for successful execution of performance tests.
- Always remember that while defining a Performance Test Strategy or Test Plan we need to focus on the test objective and scope. If our test strategy or plan is not in line with the requirements or scope then our tests are invalid.
- Try to concentrate and incorporate those metrics which are important to capture during the test run to identify any bottlenecks in the system or to see the performance of the application.
- Plan the test runs in such a way that you don’t test all the scenarios at once and crash the system. Have a number of test runs and gradually increase the scenarios and user load.
- In your approach try to add all the devices from which your application will be accessed, this usually applies to mobile devices.
- Always have a Risk and Mitigation section in your Strategy document as the requirements keep changing from time to time and this changes will have a lot of impact on the execution cycles and deadlines which have to be addressed to the client well ahead of time.
Conclusion
I’m sure that this tutorial would have briefed you the differences between a Performance Test Strategy and plan along with its contents, Approach for Mobile Application Performance Testing & Cloud application performance testing in a detailed manner with examples.
Check out our upcoming tutorial to know more about the Ways to Supercharge your Performance Testing.
=> Visit Here For Complete Performance Testing Tutorials Series