A Complete Performance Testing Guide with Examples

What is Performance Testing?

Performance Testing, also knows as ‘Perf Testing', is a type of testing performed to check how application or software performs under workload in terms of responsiveness and stability. The Performance Test goal is to identify and remove Performance bottlenecks from an application.

This test is mainly performed to check whether the software meets the expected requirements for application speed, scalability, and stability. 

In this tutorial series, we will cover complete details like – Perf Testing Types, Process, and Writing Performance Test Strategy document from scratch.

This is a detailed tutorial series you may want to bookmark!

Let's explore!

A Complete Performance Testing Guide

List of ALL the Performance Testing Tutorials in this Series:

Tutorial #1: Performance Testing Complete Guide (This Tutorial)
Tutorial #2: Difference Between Performance, Load and Stress Testing
Tutorial #3: Functional Testing Vs Performance Testing
Tutorial #4: Performance Test Plan and Test Strategy
Tutorial #5: Ways to Supercharge your Performance Testing
Tutorial #6: Cloud Performance Testing Guide
Tutorial #7: Mobile App Performance Testing Guide
Tutorial #8: How to Perform Manual Performance Testing
Tutorial #9: Website Performance Testing Tutorial
Tutorial #10: Performance Testing Companies
Tutorial #11: Performance Testing with LoadRunner (Series)

Tools:

Tutorial #12: Top Performance Testing Tools
Tutorial #13: Neoload Performance Test Tutorial
Tutorial #14: BlazeMeter Mobile Performance Test Tutorial
Tutorial #15: WAPT Load, Stress and Performance Test Tutorial
Tutorial #16: SmartMeter.io Website Performance Test Tutorial

************************************************************************

Types of Performance Testing

Performance testing types

Load Testing

Load testing is a type of performance test where the application is tested for its performance on normal and peak usage. Performance of an application is checked with respect to its response to the user request and its ability to respond consistently within an accepted tolerance on different user loads.

The key considerations are:

  1. What is the maximum load the application is able to hold before the application starts behaving unexpectedly?
  2. How much data the Database able to handle before system slows or the crash is observed?
  3. Are there any network related issues to be addressed?

Stress Testing

Stress testing is used to find ways to break the system. The test also provides the range of maximum load the system can hold.

Generally, stress testing has an incremental approach where the load is increased gradually. The test is started with a load for which the application has already been tested. Then, more load is added slowly to stress the system. The point at which we start seeing servers not responding to the requests is considered the breaking point.

The following questions are to be addressed:

  • What is the maximum load a system can sustain before it breaks down?
  • How is the system break down?
  • Is the system able to recover once it’s crashed?
  • In how many ways a system can break and which are the weak node while handling the unexpected load?

Volume Testing

Volume testing is to verify that the performance of the application is not affected by the volume of data that is being handled by the application. In order to execute a Volume Test, a huge volume of data is entered into the database. This test can be an incremental or steady test. In the incremental test, the volume of data is increased gradually.

Generally, with the application usage, the database size grows, and it is necessary to test the application against a heavy database.  A good example of this could be a website of a new school or a college having small amounts of data to store initially, but after 5-10 years, the data stores in the database of the website is much more.

Capacity Testing

=> Is the application capable of meeting business volume under both normal and peak load conditions?

Capacity testing is generally done for future prospects.  Capacity testing addresses the following:

  1. Will the application be able to support the future load?
  2. Is the environment capable of standing for the upcoming increased load?
  3. What are the additional resources required to make the environment capable enough?

Capacity testing is used to determine how many users and/or transactions a given web application will support and still meet performance. During this testing, resources such as processor capacity, network bandwidth, memory usage, disk capacity, etc. are considered and altered to meet the goal.

Online Banking is a perfect example of where capacity testing could play a major role.

Reliability/Recovery Testing

Reliability Testing or Recovery Testing – is to verify whether or not the application is able to return back to its normal state after a failure or abnormal behavior and how long does it take for it to do so (in other words, time estimation).

If an online trading site experiences a failure where the users are not able to buy/sell shares at a certain point of the day (peak hours) but are able to do so after an hour or two, we can say the application is reliable or recovered from the abnormal behavior.

Performance Test Process

Here are all the activities performed in this testing:

Performance Test workflow

#1. Requirement Analysis/Gathering

Performance team interacts with the client for identification and gathering of requirement – technical and business. This includes getting information on application's architecture, technologies and database used, intended users, functionality, application usage, test requirement, hardware & software requirements etc.

#2. POC/Tool selection

Once the key functionality is identified, POC (proof of concept – which is a sort of demonstration of the real-time activity but in a limited sense) is done with the available tools. The list of available tools depends on the cost of the tool, protocol that application is using, the technologies used to build the application, the number of users we are simulating for the test, etc. During POC, scripts are created for the identified key functionality and executed with 10-15 virtual users.

#3. Performance Test Plan & Design

Depending on the information collected in the preceding stages, test planning and designing are conducted.

Test Planning involves information on how the performance test is going to take place – test environment, workload, hardware, etc.

More on Test Strategy document below.

#4. Performance Test Development

  • Use cases are created for the functionality identified in the test plan as the scope of PT.
  • These use cases are shared with the client for their approval. This is to make sure the script will be recorded with the correct steps.
  • Once approved, script development starts with a recording of the steps in use cases with the performance test tool selected during the POC (Proof of Concepts) and enhanced by performing Correlation (for handling dynamic value), Parameterization (value substitution) and custom functions as per the situation or need. More on these techniques in our video tutorials.
  • The Scripts are then validated against different users.
  • Parallel to script creation, the performance team also keeps working on setting up of the test environment (Software and hardware).
  • Performance team will also take care of Metadata (back-end) through scripts if this activity is not taken up by the client.

#5. Performance Test Modeling


Performance Load Model is created for the test execution. The main aim of this step is to validate whether the given Performance metrics (provided by clients) are achieved during the test or not. There are different approaches to create a Load model. “Little's Law” is used in most cases.

#6. Test Execution

The scenario is designed according to the Load Model in Controller or Performance Center but the initial tests are not executed with maximum users that are in the Load model.

Test execution is done incrementally. For example: If the maximum number of users is 100, the scenarios are first run with 10, 25, 50 users and so on, eventually moving on to 100 users.

#7. Test Results Analysis

Test results are the most important deliverable for the performance tester. This is where we can prove the ROI (Return on Investment) and productivity that a performance testing effort can provide.

Some of the best practices that help the result analysis process:

a) A unique and meaningful name to every test result – this helps in understanding the purpose of the test
b) Include the following information in the test result summary:

  • Reason for the failure/s
  • Change in the performance of the application compared to the previous test run
  • Changes made in the test from the point of application build or test environment.
  • It’s a good practice to make a result summary after each test run so that analysis results are not compiled every time test results are referred.
  • PT generally requires many test runs to reach the correct conclusion.
  • It is good to have the following points in result summary:
    • Purpose of test
    • Number of virtual users
    • Scenario summary
    • Duration of test
    • Throughput
    • Graphs
    • Graphs comparison
    • Response Time
    • Error occurred
    • Recommendations

#8. Report

Test results should be simplified so the conclusion is clearer and should not need any derivation. Development Team needs more information on analysis, comparison of results, and details of how the results were obtained.

The test report is considered to be good if it is brief, descriptive and to the point.

How to Write Performance Test Strategy Document?

This tutorial will explain how to write a sample Performance Test Strategy for a Messaging Application.

Remember, that this is just an example and the requirements will differ from one client to another, we will also get to know the best practices for performance testing in this tutorial.

Performance Test Strategy

Sample Performance Test Strategy Template

About ABC chat Application – Let’s assume that this is a chat workbench that is used in a company by their customer support agent, this chat application uses XMPP protocol i.e, Extensible Messaging and Presence Protocol and Open fire server for sending and receiving Instant messages.

Some enhancements have been made to this existing chat client like Remote PC control, PC diagnosis, Repair tools, Online chat etc., so this performance Test strategy is a sample of such application.

For this application let’s assume that the project team has decided to use JMeter for Performance Testing and JIRA for defect tracking.

The first page of the Performance Test Strategy document should contain the Tittle of the document and the copyrights of the company.

The second page should contain Document Control which includes, Document Version history, Reviewers & Approvers list and Contributors list.

The third page should contain the Table of contents, followed by the below topics.

Introduction

The purpose of this document is to define/explain how Performance Testing will be performed on the ABC chat application for the current and future state.

ABC chat application is an in-house remote support Agent workbench. This workbench will be used to fulfill customer requests. This Workbench has capabilities such as Online chat, Customer Identification, Remote PC control, PC diagnosis and Repair tools.

Objective

The key objectives of performance testing are as follows:

  • To gain the confidence that the changes to the existing chat application are in line with the defined Service Level Agreement.
  • To ensure that the application performance, service availability, and the stability of the application are not impacted as a result of the new enhancements.
  • Transaction Response Times remain within the acceptable tolerance over the increasing Load profile.
  • JVMs show stable memory usage over the increasing load profiles.

The picture given below clearly explains Performance Testing & Optimization process:

Performance Testing & Optimization process

Architecture

You need to incorporate the architecture diagram of your project in this session.

Scope

Inscope

Below is the performance testing scope for ABC chat workbench:

  • Knowledge acquisition of the key business transactions and build load distribution after a detailed study of the system.
  • Identify the critical scenarios for performance testing with assistance from different project tracks.
  • Use the previous release results as a baseline for future releases.
  • Verify and validate the performance test environment and the Performance/Load test tool infrastructure for any additional Agent Machines.
  • Preparation of performance test scripts using JMeter for the identified scenarios that mimic the identified peak load.
  • Setup performance monitoring on the servers for monitoring the test in order to identify the bottlenecks during the test execution phase.
  • Publish Performance test results.
  • Coordinate with various stakeholders to resolve the identified performance issues.
  • Baseline the performance level for future releases.

Out of Scope

  • Functional Testing, UAT, System Testing & Security Testing.
  • Performance testing/monitoring of any third party interfaces.
  • Performance Tuning. (Most of the times Tuning is done by a different team, if in case you have performance engineers to tune the system then you can add this in the Inscope).
  • Code profiling / Hardware sizing / Capacity planning.
  • Security / Vulnerability testing / UAT/ White box testing.
  • Data generation for Performance Testing.
  • Non-functional tests (E.g. failover, disaster recovery, back-up, usability) other than the performance tests.
  • Testing of any mobile solution.
  • Third Party Application Performance Testing & Tuning.
  • Realization of performance recommendations, Application code changes and the vendor supported products/server configuration changes will be out of scope from the Performance Team perspective.
  • Infrastructure Support / Build Deployment/ Environment Readiness/ Database Restore/ Network Support etc.

Approach

Performance testing for ABC chat will be conducted using Jmeter by writing custom XMPP plugins which use smack library for XMPP connections. These libraries are used to setup connection, login and send chat messages to the XMPP server.

These libraries are bundled into a jar file which is deployed into the Jmeter and are designed based on the scenarios to be tested. The Jmeter Work Bench is installed in the local machine which connects to the JMeter server which has the Load Generators to generate the required load on the Chat server system to monitor the system behavior.

Test scenario will be scripted using the JMeter tool. The scripts would be customized as required. The schedule will be created with the required ramp up to simulate the real world scenarios.

The test scenario would be broken up and measured in the below aspects:

a) Baseline Test: To run each scenario with 1 Vuser and multiple iterations in order to identify whether the application performance meets the business Service Level Agreement or not.

b) Base Load Test: To meet the Business Benchmark under load test, the Performance Testing team will perform a baseload test which will help to identify any system performance issues with increasing load and creates the baseline for the next level of performance testing.

c) Peak Load / Scalability Test: Performance Testing team will perform multiple tests with increasing Vusers to meet the expected load and also to measure the application performance to establish the performance curve and identify whether the deployment can support the service level agreements under the peak user load.

It helps in tuning or capacity planning of the individual Java virtual machines (JVM), the total number of required JVMs, and the processors. This will be achieved by increasing the no of Vusers to 50%, 75%, 100% and 125% of peak capacity.

d) Endurance Test: Performance Testing team will run this test for a period of 8 Hours / 16 Hours /24 hours to identify memory leaks, performance issues over time, and overall system stability. During endurance tests, the Performance Testing team monitors the key performance indicators, such as transaction response times and the stability of memory usage.

System resources like CPU, Memory, and IO need to be monitored with the help of the project team.

The Performance test environment is assumed to be a replica of the production environment. The tests will be run with an incremental load to identify where the application fails.

Performance Test Scenarios

Include the excel with the set of scenarios.

For Example:

Scenario 1: To validate the Agent and customer chat for X no. of concurrent sessions.

Performance Test Types

The table given below explains the various types of Performance Tests along with their objectives.

Test TypeObjective
Baseline TestEstablish the best performance under specific volumes which will be used as a reference for subsequent measurements.
Load TestMeasure the system performance under anticipated peak production load.
Endurance TestMeasuring the system stability under high volume for an extended period.
Stress TestMeasure the system performance under unfavorable conditions.

Performance Metrics

Client-side Metrics
S.NoMetricDescriptionFormat
1Transaction Response TimeResponse time of pages during the steady state of the performance testGraph
2ThroughputThe amount of data that the VUsers received from the server over timeGraph
3Hits/secondThe number of HTTP requests made by VUsers to the Web server during the scenario runGraph
4Number of Passed/Failed Transactions

Total number of transactions that Passed and Failed during the test executionExcel
5Transaction Error RateThe Percentage of transactions that failed during the test executionGraph
System & Network Performance Metrics

System & Network Performance Metrics

Performance Testing Activities & Deliverables

Performance Testing Activities

Test Data

It is being assumed that the Performance environment data will be a copy of the production data and the required Test data will be provided by the project team.

Entry & Exit Criteria

  • Access to all the applications in the environment.
  • Environment readiness complete.
  • Performance Test Data readiness.

Entry & Exit Criteria

Defect Management

  • The defect management module in JIRA will be used in the project for defect logging and for tracking to closure.
  • Identification of defects that are found during the test execution phase will be captured in JIRA and these defects will be resolved by the development team according to the below severities.
  • Defect review meetings would be held on a daily basis with the participation from the testing, development, Quality Analysts, and business teams.
  • The criteria to fix defects would get stringent as the project approaches the Go Live date. Guidelines for defect fix criteria to be published in defect review meetings.

Defect Severity Definition

The definitions of severity codes are as follows:

SeverityDescription for Development and Enhancement Problems
BlockerSystem error, show stopper, Network issues
CriticalSystem errors, no clear workaround, interruption or missing business functionality

MajorA serious problem was detected for which the workaround exists that might not be clear to all the users, however, product should not be released without fixing
MediumProblem exists with easy/simple work around but this type of defect may be released upon approval by Business and/or Project Manager
LowCosmetic issues that do not interfere with business functionality or other intermittent problems that are not reproducible every time

Testing Tools & Techniques

ToolsPurpose
JmeterTo verify the Load and Performance of the ABC Chat application.

Suspension and Resumption Criteria

Given below are the critical suspension and resumption criteria which will impact the testing activities:

SuspensionImpactResumption
Environment not set upTesting cannot proceed

Environment readiness.
Application found to be unstableTesting cannot proceed.

Issue resolved
Test Data not availableTesting cannot proceed.

Test Data ready

Test Deliverables

The Performance test deliverables include:

  • Performance Testing Strategy
  • Performance Requirements Document
  • Performance Test Scenario Document
  • Performance Test Scripts
  • Performance Test Results

Roles & Responsibilities

Roles & Responsibilities are clearly explained in the table given below.

Roles & Responsibilities

Potential Risks & Mitigation Plan

S.NoRiskProbabilityImpactMitigation PlanOwner
1Test Data unavailability for performance load test executionsHHEstimated dates for the performance test executions should be reviewed and updated. Functional/Dev team support required for data gathering.--
2Environmental IssuesLMRe prioritize Deliverables--
3Change in Functionality/design during performance test execution

MHThis requires rework on the performance test scenarios--
4Extra performance runs to troubleshoot performance issuesMHPerformance testing schedules would be modified and updated to the product team.--
5Estimations are prepared based on 1 bug fix build for performance. Multiple bug fix builds will delay test cycles and eventually it depends on when the next build will be available for rerun.HHRe prioritize the performance test execution cycles.--
6Hardware AvailabilityMHSchedule start date would be moved accordingly.--

Assumptions

  • Performance Test Environment will be a replica of the production architecture landscape. (i.e. correct Hardware, Software, Interfaces, Integration Layers etc).
  • Performance scripts will be designed based on the critical flows for which the usage is high.
  • All Infrastructure Issues should be resolved before the beginning of Performance testing. Any system configuration changes made later will invalidate the Test results.
  • An application is stable and ready to use in the Performance test environment.
  • Necessary hardware and software resources (like load generator machines/software, controller/agent machines) are made available.
  • Any changes to the scope will go through a change control process and the performance testing team will assess the impact of timelines and resources.
  • Respective Servers are expected to handle the load.
  • Application trace logs have to be enabled for the supporting systems for monitoring purposes.

Dependencies

  • Availability of the Performance test environment which is a replica of the production architecture landscape.
  • Support required from various Functional, Development, Database and Infrastructure teams during the test preparation and execution stages.
  • No code changes are implemented during the entire Performance testing phase as time is very limited.
  • In the event of unforeseen issues that lead to restrictions within the timelines, if timelines do not allow for all the test scopes to be met within the original milestone dates support is available from the Release Managers, to provide a scoping and prioritization decision.
  • Application Business Users / Subject Matter Experts will be made available for functional clarifications, and business transactions sign-off.
  • ABC chat Program Manager will review and sign-off.

Abbreviations

AbbreviationDescription
DBDatabase
HttpHyper Text Transfer Protocol
JDBCJava Database Connectivity
QAQuality Assurance
SLAService Level Agreement
SMESubject Matter Expert
UATUser Acceptance Testing

By now you must have clearly understood how to write an effective Performance Test Strategy for a Messaging application.

Best Practices for Realistic Performance Testing

In order to complete a performance test project successfully, we need to ensure that we are doing it in the right way from the planning stage ie planning, development, execution and analysis.

Let’s take a look at each stage in detail in order to conduct performance testing effectively.

#1) Planning

(i) Try to identify the most common workflows i.e the business scenarios which have to be tested. If the application is an existing one, then check the server logs to understand the most frequently accessed scenarios. If the application is new than talk to the project management team to understand the major business flow.

(ii) Plan the Load test in such a way that you cover a broad range of workflows like light usage, medium usage and peak loads.

(iii) You need to perform many cycles of Load test, so try to create a framework so that you can use the same scripts again and again. Also, try to have a backup of the scripts.

(iv) Try to analyze how long a test has to run, is it one hour? 8 hours? A day or a week? Usually, long duration tests will uncover many major defects such as OS bugs, Memory leaks etc.

(v) If your organization is using any APM (Application Monitoring Tool), then you can include it during the test runs so that you can easily identify the performance issues and identify the root cause more easily.

#2) Development

(i) While developing the scripts i.e recording, try to give a more meaningful transaction name based on the business flow names that are mentioned in the plan.

(ii) Don’t record any third party applications and if it gets recorded, try to filter it out while enhancing the scripts.

(iii) Not all the dynamic values can be correlated using Autocorrelation feature in the tool, so try to do a manual correlation to avoid errors.

(iv) Try to design your performance tests in such a way that you are hitting the backend of the application and not just the cache server.

#3) Execution

(i) Make sure to run the tests in production like environment, including factors like SSL, Load Balancer, and Firewalls. This is necessary to simulate a realistic load on the system.

(ii) Try to create a workload which is very realistic, you can get this by checking the server logs if it is an existing application and if it is a new application you need to get this info from the business team. Remember that workload is very important for conducting successful performance tests.

(iii) Never come to a conclusion by running tests with half the production size environment, it is always advised to conduct tests in an environment which is just the same as production.

(iv) While executing long run tests try to watch the run at frequent intervals in order to make sure that the test is running smoothly.

#4) Analysis

(i) Try to analyze the application by adding a few important counters first, when a bottleneck is found then try to add additional counters with respect to the bottleneck. This, in turn, will help in finding the issue more easily.

(ii) An application can fail for many reasons like it can fail to respond to a request, respond with an error code, fail your validation logic or responding too slowly. So try to look into all these before coming to a conclusion.

Conclusion

I'm sure that this tutorial would have given you immense knowledge on Performance test and how to write a Performance Test Strategy Document with detailed examples.

In our upcoming tutorial, we will learn the differences between Performance, Load and Stress Testing in detail.

Also Check => Free LoadRunner In-depth Training Series

NEXT Tutorial

Recommended Reading


72 thoughts on “A Complete Performance Testing Guide with Examples”

  1. Hi Vijay sir, this article is really helpful for beginners . I am planing to switch my carrier in performance testing and I am having an 1.7 years of experience in manual testing . Can you please guide that this much of experience is sufficient for switching into performance testing.

    Sir please guide me.

  2. Hi Chetan Kaushal,
    i liked your article.I want to learn LR correlation topic and analysis part in depth..
    Are you providing any online training /instuite on this.if not,can you please suggest any websites where i can learn this indepth

  3. Hi Chetan,

    I am 54 years old and have bachelor degree in electrical engineering. I have been thinking for a long to switch my career in IT. I explored a field of my interest and Load Runner seems me good to get into it. I download the tutorials from HP site and also got installed Load runner 12.0.
    The problem I am facing is connecting with the web tour application server. The HP has a comment if the file is not saved in default the web application connection will not be possible. I have not understood it if the load runner has been installed on my machine and appears in my all programs button how this can be not in default files.
    Can you please help me to get it installed properly so that I could also take advantage of your lectures to learn it with practice?

  4. Hi,

    I have over 2.5 years exp in performance testing using load runner and currently working as a Java dev from past 1.3 yrs.

    Total exp:2.5+1.3 yrs.

    Which is a better carrier option Perftesting or Java Dev.
    Please advice.

  5. Hi Vijay Mehta,

    Load Runner is just a tool to implement Performance testing. Learning just Load Runner will not take you any where. You need to read and know about Performance testing, HTTP concepts, World Wide Web Concepts, Network concepts and general concpets on how web applications work. Having a fair understanding of all these concepts is more essential than knowing load runner. Once you know these concepts, you can use any of the many load testing tools and then learn how to implement these concepts in a particular tool.

    And to learn / practise load runner, you dont need the webtours application that HP provides. As long as you have network connectivity on your application, you can practise script creating on any of the applications available online.

    Ofcourse, you cannot execute load tests on those applications, if you do they may sue you.

    One of the best internet application to practise Load Runner is http://www.demo.opencart.com

    Again, learn the concepts first then tools. Learning just load runner without the basic foundation in Performance testing is like learning to user movie cameras when what you want to actually do is Movie Direction..

    Camera is just a tool… story, screenplay, direction is the content, underlying foundation… !!

  6. Hi,
    Thanks for the good info.Can you please let us know if we want to become master in performance testing in load runner what should I do. As of now I just have vugen knowledge of writing scripts and running but very minimal knowledge in analysis

  7. Hello Lavanya, recently I have learned performance testing online at itelearn.

    Really this article is very impressive. Keep it up .

  8. Hi SoftwareTestingHelp Team,

    I was looking for an article on graph databases- Neo4j.
    Do we have it yet ? I searched on this website but could not find.I would love to have it here as you explain the topics so nicely.
    Please reply.
    Thanks.

Leave a Comment