A Complete Performance Testing Guide With Examples

What is Performance Testing?

Performance Testing also knows as ‘Perf Testing’, is a type of testing performed to check how application or software performs under workload in terms of responsiveness and stability. The Performance Test goal is to identify and remove performance bottlenecks from an application.

This test is mainly performed to check whether the software meets the expected requirements for application speed, scalability, and stability. 

In this tutorial series, we will cover complete details like – Perf Testing Types, Process, and Writing Performance Test Strategy document from scratch.

This is a detailed tutorial series you may want to bookmark!

Let’s explore!

List of ALL the Performance Testing Tutorials in this Series:

Tutorial #1: Performance Testing Complete Guide (This Tutorial)
Tutorial #2: Difference Between Performance, Load and Stress Testing
Tutorial #3: Functional Testing Vs Performance Testing
Tutorial #4: Performance Test Plan and Test Strategy
Tutorial #5: Ways to Supercharge your Performance Testing
Tutorial #6: Cloud Performance Testing Guide
Tutorial #7: Mobile App Performance Testing Guide
Tutorial #8: How to Perform Manual Performance Testing
Tutorial #9: Website Performance Testing Tutorial
Tutorial #10: Performance Testing Companies
Tutorial #11: Performance Testing with LoadRunner (Series)

Tools:

Tutorial #12: Top Performance Testing Tools
Tutorial #13: Neoload Performance Test Tutorial
Tutorial #14: BlazeMeter Mobile Performance Test Tutorial
Tutorial #15: WAPT Load, Stress and Performance Test Tutorial
Tutorial #16: SmartMeter.io Website Performance Test Tutorial

Types Of Performance Testing

Load Testing

Load Testing is a type of performance test where the application is tested for its performance on normal and peak usage. The performance of an application is checked with respect to its response to the user request and its ability to respond consistently within an accepted tolerance on different user loads.

The key considerations are:

1. What is the maximum load the application is able to hold before the application starts behaving unexpectedly?
2. How much data the Database able to handle before the system slows or the crash is observed?
3. Are there any network related issues to be addressed?

Stress Testing

Stress Testing is used to find ways to break the system. The test also provides the range of maximum load the system can hold.

Generally, Stress Testing has an incremental approach where the load is increased gradually. The test is started with a load for which the application has already been tested. Then, more load is added slowly to stress the system. The point at which we start seeing servers not responding to the requests is considered the breaking point.

The following questions are to be addressed:

• What is the maximum load a system can sustain before it breaks down?
• How is the system break down?
• Is the system able to recover once it’s crashed?
• In how many ways a system can break and which are the weak node while handling the unexpected load?

Volume Testing

Volume Testing is to verify that the performance of the application is not affected by the volume of data that is being handled by the application. In order to execute a Volume Test, a huge volume of data is entered into the database. This test can be an incremental or steady test. In the incremental test, the volume of data is increased gradually.

Generally, with the application usage, the database size grows, and it is necessary to test the application against a heavy database.  A good example of this could be a website of a new school or a college having small amounts of data to store initially, but after 5-10 years, the data stores in the database of the website is much more.

Capacity Testing

=> Is the application capable of meeting business volume under both normal and peak load conditions?

Capacity Testing is generally done for future prospects.  Capacity Testing addresses the following:

1. Will the application be able to support the future load?
2. Is the environment capable of standing for the upcoming increased load?
3. What are the additional resources required to make the environment capable enough?

Capacity Testing is used to determine how many users and/or transactions a given web application will support and still meet performance. During this testing, resources such as processor capacity, network bandwidth, memory usage, disk capacity, etc. are considered and altered to meet the goal.

Online Banking is a perfect example of where capacity testing could play a major role.

Reliability/Recovery Testing

Reliability Testing or Recovery Testing – is to verify whether or not the application is able to return back to its normal state after a failure or abnormal behavior and how long does it take for it to do so (in other words, time estimation).

If an online trading site experiences a failure where the users are not able to buy/sell shares at a certain point of the day (peak hours) but are able to do so after an hour or two, we can say the application is reliable or recovered from the abnormal behavior.

Performance Test Process

Here are all the activities performed in this testing:

#1) Requirement Analysis/Gathering

The performance team interacts with the client for identification and gathering of requirements – technical and business. This includes getting information on the application’s architecture, technologies, and database used, intended users, functionality, application usage, test requirement, hardware & software requirements, etc.

#2) POC/Tool selection

Once the key functionality is identified, POC (Proof Of Concept – which is a sort of demonstration of the real-time activity but in a limited sense) is done with the available tools.

The list of available tools depends on the cost of the tool, protocol that application is using, the technologies used to build the application, the number of users we are simulating for the test, etc. During POC, scripts are created for the identified key functionality and executed with 10-15 virtual users.

#3) Performance Test Plan & Design

Depending on the information collected in the preceding stages, test planning and designing are conducted.

Test Planning involves information on how the performance test is going to take place – test environment, workload, hardware, etc.

More on the Test Strategy document below.

#4) Performance Test Development

• Use cases are created for the functionality identified in the test plan as the scope of PT.
• These use cases are shared with the client for their approval. This is to make sure the script will be recorded with the correct steps.
• Once approved, script development starts with a recording of the steps in use cases with the performance test tool selected during the POC (Proof of Concepts) and enhanced by performing Correlation (for handling dynamic value), Parameterization (value substitution) and custom functions as per the situation or need. More on these techniques in our video tutorials.
• The Scripts are then validated against different users.
• Parallel to script creation, the performance team also keeps working on setting up the test environment (Software and hardware).
• The performance team will also take care of Metadata (back-end) through scripts if this activity is not taken up by the client.

#5) Performance Test Modeling

Performance Load Model is created for the test execution. The main aim of this step is to validate whether the given Performance metrics (provided by clients) are achieved during the test or not. There are different approaches to create a Load model. “Little’s Law” is used in most cases.

#6) Test Execution

The scenario is designed according to the Load Model in Controller or Performance Center but the initial tests are not executed with maximum users that are in the Load model.

Test Execution is done incrementally. For Example, If the maximum number of users is 100, the scenarios are first run with 10, 25, 50 users and so on, eventually moving on to 100 users.

#7) Test Results Analysis

Test results are the most important deliverable for the performance tester. This is where we can prove the ROI (Return on Investment) and productivity that a performance testing effort can provide.

Some of the best practices that help the Result Analysis process:

1. A unique and meaningful name to every test result – this helps in understanding the purpose of the test.
2. Include the following information in the test result summary:

• Reason for the failure/s
• Change in the performance of the application compared to the previous test run
• Changes made in the test from the point of application build or test environment.
• It’s a good practice to make a result summary after each test run so that analysis results are not compiled every time test results are referred.
• PT generally requires many test runs to reach the correct conclusion.
• It is good to have the following points in result summary:
  • Purpose of test
  • Number of virtual users
  • Scenario summary
  • Duration of test
  • Throughput
  • Graphs
  • Graphs comparison
  • Response Time
  • Error occurred
  • Recommendations

#8) Report

Test results should be simplified so the conclusion is clearer and should not need any derivation. Development Team needs more information on analysis, comparison of results, and details of how the results were obtained.

The test report is considered to be good if it is brief, descriptive and to the point.

How To Write Performance Test Strategy Document?

This tutorial will explain how to write a sample Performance Test Strategy for a Messaging Application.

Remember, that this is just an example and the requirements will differ from one client to another, we will also get to know the best practices for Performance Testing in this tutorial.

Sample Performance Test Strategy Template

About ABC chat Application – Let’s assume that this is a chat workbench that is used in a company by their customer support agent, this chat application uses XMPP protocol i.e, Extensible Messaging and Presence Protocol and Open fire server for sending and receiving Instant messages.

Some enhancements have been made to this existing chat client like Remote PC control, PC diagnosis, Repair tools, Online chat, etc., so this performance Test strategy is a sample of such applications.

For this application let’s assume that the project team has decided to use JMeter for Performance Testing and JIRA for defect tracking.

The first page of the Performance Test Strategy document should contain Title of the Document and the Copyrights of the Company.

The second page should contain Document Control which includes, Document Version history, Reviewers & Approvers list and Contributors list.

The third page should contain the Table of contents, followed by the below topics.

#1) Introduction

The purpose of this document is to define/explain how Performance Testing will be performed on the ABC chat application for the current and future state.

ABC chat application is an in-house remote support Agent workbench. This workbench will be used to fulfill customer requests. This Workbench has capabilities such as Online chat, Customer Identification, Remote PC control, PC diagnosis, and repair tools.

Objective

The key objectives of Performance Testing are as follows:

• To gain the confidence that the changes to the existing chat application are in line with the defined Service Level Agreement.
• To ensure that the application performance, service availability, and the stability of the application are not impacted as a result of the new enhancements.
• Transaction Response Times remain within the acceptable tolerance over the increasing Load profile.
• JVMs show stable memory usage over the increasing load profiles.

The below picture clearly explains Performance Testing & Optimization process:

Architecture

You need to incorporate the architecture diagram of your project in this session.

#2) Scope

In Scope

Below is the Performance Testing scope for ABC chat workbench:

• Knowledge acquisition of the key business transactions and build load distribution after a detailed study of the system.
• Identify the critical scenarios for performance testing with assistance from different project tracks.
• Use the previous release results as a baseline for future releases.
• Verify and validate the performance test environment and the Performance/Load test tool infrastructure for any additional Agent Machines.
• Preparation of performance test scripts using JMeter for the identified scenarios that mimic the identified peak load.
• Setup performance monitoring on the servers for monitoring the test in order to identify the bottlenecks during the test execution phase.
• Publish Performance test results.
• Coordinate with various stakeholders to resolve the identified performance issues.
• Baseline the performance level for future releases.

Out of Scope

• Functional Testing, UAT, System Testing & Security Testing.
• Performance testing/monitoring of any third-party interfaces.
• Performance Tuning. (Most of the times Tuning is done by a different team, if in case you have performance engineers to tune the system then you can add this in the Inscope).
• Code profiling / Hardware sizing / Capacity planning.
• Security / Vulnerability testing / UAT/ White box testing.
• Data generation for Performance Testing.
• Non-functional tests (For Example, failover, disaster recovery, back-up, usability) other than the performance tests.
• Testing of any mobile solution.
• Third-Party Application Performance Testing & Tuning.
• Realization of performance recommendations, Application code changes and the vendor-supported products/server configuration changes will be out of scope from the Performance Team perspective.
• Infrastructure Support / Build Deployment/ Environment Readiness/ Database Restore/ Network Support etc.

#3) Approach

Performance testing for ABC chat will be conducted using Jmeter by writing custom XMPP plugins that use a smack library for XMPP connections. These libraries are used to set up connections, login and send chat messages to the XMPP server.

These libraries are bundled into a jar file that is deployed into the Jmeter and is designed based on the scenarios to be tested. The Jmeter Work Bench is installed in the local machine which connects to the JMeter server which has the Load Generators to generate the required load on the Chat server system to monitor the system behavior.

The test scenario will be scripted using the JMeter tool. The scripts would be customized as required. The schedule will be created with the required ramp-up to simulate the real-world scenarios.

The Test Scenario would be broken up and measured in the below aspects:

a) Baseline Test: To run each scenario with 1 Vuser and multiple iterations in order to identify whether the application performance meets the business Service Level Agreement or not.

b) Base Load Test: To meet the Business Benchmark under load test, the Performance Testing team will perform a baseload test which will help to identify any system performance issues with increasing load and creates the baseline for the next level of performance testing.

c) Peak Load / Scalability Test: Performance Testing team will perform multiple tests with increasing Vusers to meet the expected load and also to measure the application performance to establish the performance curve and identify whether the deployment can support the service level agreements under the peak user load.

It helps in tuning or capacity planning of the individual Java virtual machines (JVM), the total number of required JVMs, and the processors. This will be achieved by increasing the no of Vusers to 50%, 75%, 100% and 125% of peak capacity.

d) Endurance Test: Performance Testing team will run this test for a period of 8 Hours / 16 Hours /24 hours to identify memory leaks, performance issues over time, and overall system stability. During endurance tests, the Performance Testing team monitors the key performance indicators, such as transaction response times and the stability of memory usage.

System resources like CPU, Memory, and IO need to be monitored with the help of the project team.

The Performance test environment is assumed to be a replica of the production environment. The tests will be run with an incremental load to identify where the application fails.

Performance Test Scenarios

Include the excel with the set of scenarios.

For Example,

Scenario 1: To validate the Agent and customer chat for X no. of concurrent sessions.

Performance Test Types

The table given below explains the various types of Performance Tests along with their objectives.

Performance Metrics

• Client-side Metrics

• System & Network Performance Metrics

Performance Testing Activities & Deliverables

#4) Test Data

It is being assumed that the Performance environment data will be a copy of the production data and the required test data will be provided by the project team.

#5) Entry & Exit Criteria

• Access to all the applications in the environment.
• Environment readiness complete.
• Performance Test Data readiness.

#6) Defect Management

• The Defect Management module in JIRA will be used in the project for defect logging and for tracking to closure.
• Identification of defects that are found during the test execution phase will be captured in JIRA and these defects will be resolved by the development team according to the below severities.
• Defect review meetings would be held on a daily basis with the participation from the testing, development, Quality Analysts, and business teams.
• The criteria to fix defects would get stringent as the project approaches the Go Live date. Guidelines for defect fix criteria to be published in defect review meetings.

Defect Severity Definition

The definitions of severity codes are as follows:

#7) Testing Tools & Techniques

#8) Suspension and Resumption Criteria

Given below are the Critical Suspension and Resumption criteria which will impact the testing activities:

#9) Test Deliverables

The Performance Test Deliverables include:

• Performance Testing Strategy
• Performance Requirements Document
• Performance Test Scenario Document
• Performance Test Scripts
• Performance Test Results

#10) Roles & Responsibilities

Roles & Responsibilities are clearly explained in the table given below.

#11) Potential Risks & Mitigation Plan

#12) Assumptions

• Performance Test Environment will be a replica of the product architecture landscape. (i.e. correct Hardware, Software, Interfaces, Integration Layers, etc).
• Performance scripts will be designed based on the critical flows for which the usage is high.
• All Infrastructure Issues should be resolved before the beginning of Performance testing. Any system configuration changes made later will invalidate the test results.
• An application is stable and ready to use in the Performance test environment.
• Necessary hardware and software resources (like load generator machines/software, controller/agent machines) are made available.
• Any changes to the scope will go through a change control process and the performance testing team will assess the impact of timelines and resources.
• Respective Servers are expected to handle the load.
• Application trace logs have to be enabled for the supporting systems for monitoring purposes.

#13) Dependencies

• Availability of the Performance test environment which is a replica of the product architecture landscape.
• Support required from various Functional, Development, Database and Infrastructure teams during the test preparation and execution stages.
• No code changes are implemented during the entire Performance testing phase as time is very limited.
• In the event of unforeseen issues that lead to restrictions within the timelines, if timelines do not allow for all the test scopes to be met within the original milestone dates support is available from the Release Managers, to provide a scoping and prioritization decision.
• Application Business Users / Subject Matter Experts will be made available for functional clarifications, and business transactions sign-off.
• ABC chat Program Manager will review and sign-off.

#14) Abbreviations

By now you must have clearly understood how to write an effective Performance Test Strategy for a Messaging application.

Best Practices For Realistic Performance Testing

In order to complete a Performance Test project successfully, we need to ensure that we are doing it in the right way from the planning stage ie planning, development, execution and analysis.

Let’s take a look at each stage in detail in order to conduct Performance Testing effectively.

#1) Planning

• Try to identify the most common workflows i.e the business scenarios which have to be tested. If the application is an existing one, then check the server logs to understand the most frequently accessed scenarios. If the application is new than talk to the project management team to understand the major business flow.
• Plan the Load test in such a way that you cover a broad range of workflows like light usage, medium usage and peak loads.
•  You need to perform many cycles of the Load Test, so try to create a framework so that you can use the same scripts again and again. Also, try to have a backup of the scripts.
• Try to analyze how long a test has to run, is it one hour? 8 hours? A day or a week? Usually, long-duration tests will uncover many major defects such as OS bugs, Memory leaks, etc.
• If your organization is using any APM (Application Monitoring Tool), then you can include it during the test runs so that you can easily identify the performance issues and identify the root cause more easily.

#2) Development

• While developing the scripts i.e recording, try to give a more meaningful transaction name based on the business flow names that are mentioned in the plan.
• Don’t record any third-party applications and if it gets recorded, try to filter it out while enhancing the scripts.
• Not all the dynamic values can be correlated using Autocorrelation feature in the tool, so try to do a manual correlation to avoid errors.
• Try to design your performance tests in such a way that you are hitting the backend of the application and not just the cache server.

#3) Execution

• Make sure to run the tests in a production-like environment, including factors like SSL, Load Balancer, and Firewalls. This is necessary to simulate a realistic load on the system.
• Try to create a workload which is very realistic, you can get this by checking the server logs if it is an existing application and if it is a new application you need to get this info from the business team. Remember that workload is very important for conducting successful performance tests.
• Never come to a conclusion by running tests with half the production size environment, it is always advised to conduct tests in an environment which is just the same as production.
• While executing long-run tests try to watch the run at frequent intervals in order to make sure that the test is running smoothly.

#4) Analysis

• Try to analyze the application by adding a few important counters first, when a bottleneck is found then try to add additional counters with respect to the bottleneck. This, in turn, will help in finding the issue more easily.
• An application can fail for many reasons like it can fail to respond to a request, respond with an error code, fail your validation logic or responding too slowly. So try to look into all these before coming to a conclusion.

Conclusion

I’m sure that this tutorial would have given you immense knowledge on Performance tests and how to write a Performance Test Strategy Document with detailed examples.

In our upcoming tutorial, we will learn the differences between Performance, Load and Stress Testing in detail.

Also, Check => Free LoadRunner In-depth Training Series

NEXT Tutorial