What is Longevity Testing? How to Catch the Bugs Before the Customer Finds It

This article explains the meaning of “Longevity Testing” and how it helps to assess the stability of the System or the Product and reduce the defects found by the customer i.e. “Catch the bugs in-house before the customer finds it”.

By the end of this article QA Managers, Leads and Testers will have a fair knowledge about:  

• What is Longevity Testing?
• Why is Longevity Testing required?
• Planning and Executing Longevity Tests
• What are the Pros and Cons of Longevity Testing?

What is Longevity Testing?

Longevity Testing is a Testing Activity:

• To validate system and product stability and serviceability features over a longer period against appropriate load and stress conditions with real-time traffic and applications
• Reduce the occurrence of defects surfacing at the Customer site

Flow diagram for handling Customer reported issues (Fig. 1)

Background to Longevity Testing

#1) Usually, in the first few weeks of the Product deployment or after an upgrade to the latest Software release at the customer site, everything runs well. However, over a period of a few weeks, the customer started reporting issues.

#2) Many of the issues may have simple features as they are reported by the customer and are not easily reproducible in-house. They need a lot of time and careful analysis by the Expert Team across the spectrum. Hint: Time=$$$!!!

#3) One or more of the following happens when the customer(s) finds the defect (Fig. 1)

• Severity of the defect will have a direct impact on the Customer’s business i.e. $$$
• Any service request from the Technical Support Center is costing $$$ to the Product Engineering Organization
• Seldom the issues raised by the customer are resolved by the front end Technical Support Team
• Such requests or tickets will be escalated to our Escalation Support Team
• Customer Ticket Escalation will cost more $$$ to the Organization
• If the Escalation team is unable to resolve the issue, it will now have to involve the Engineering Team (Development and QA)
• By now the cost $$$ to resolve the issue would have also raised substantially
• Longer the defect resolution, higher the probability of dissatisfied customer(s) who would not give repeat orders and the worst scenario is when the customer decides to move to a competitor’s solution at an opportune time. However, in both cases it is a revenue loss to any Product Engineering Organization

4) The higher percentage of such issues reported by a customer(s) are related to typical System or Product stability in combination with customer topology, infrastructure, traffic, and application specific.

Why is Longevity Testing Required?

1) Any ‘Defect’ that arises out of Customer reported that the issue is usually a Test Escape.

2) Any such defects cost bottom-line $$$ to the Customer as well as the Engineering Organization that provides solutions and services to the customers.

3) In a normal scenario, the defect should have been noticed internally during various testing cycles including Regression Testing by one or more testers from the Testing Team depending upon the complexity of the issue.

4) Most importantly, such defects arising out of customer reported issues also point out an appropriate test scenario or a test case from being missed out at the point of Test Plan execution.

5) Many of the Testers must have experienced a particular feature failing at the customer site but passing in-house on various Testbeds like

• Feature
• Regression
• Load
• Stress
• Performance
• System
• Solution
• Alpha
• Beta

6) Key observations to be considered –

• During any software release cycle, System Under Test (SUT) or Device Under Test (DUT) in all Testbeds are frequently soft or hard rebooted for want of things like loading of new code drop, bug verification etc.
• Even Automated Regression Test suites usually reboot or reset the SUT or DUT post execution of a particular test case script or a series of test case scripts
• So the SUT or DUT is not running long enough without a soft or hard reboot
• Whereas the situation is entirely different at the customer site. The customer cannot afford to keep rebooting the System frequently thereby resulting in productivity disruptions
• Customers follow a proven practice where they announce a proper maintenance window to the intended audience and then carry out Software upgrades, Hardware replacements, etc.
• Such maintenance windows can be for a specific duration from Quarterly to Yearly depending on the internal guidelines and procedures of the Customer’s Organization
• In reality, the actual health picture of the System or the Product at the customer site is entirely different to that of Testbeds during a given Software Release cycle in any Product Engineering Organization
• Many customers are also looking for an authorized quality document having passed particular Vertical Model Testing, especially Financial, healthcare and Federal Verticals

Considering a few Test gaps as mentioned above =>

• It is apparent that the System or the Product should undergo longer duration of tests or Longevity Tests with end-to-end scenarios mimicking Customer Site or verticals
• Longer duration can be 72-720 hrs. (3-30 days) or appropriate duration based on EFD and CFD data and specific customer cases
• The recommended practice for QA Managers, Leads and Testers is to carry out Longevity Testing as a separate activity in a given Software Release cycle
• Net-Net and Longevity Testing is very much relevant to the stability of the System or the Product as it has a direct relationship to bottom-line $$$ of the Organization

Planning and Executing Longevity Tests

It is important that QA Managers, Leads, and Testers include Longevity Testing as part of their overall Test Strategy.

Planning

• Engineering Organizations carry out in-house Test Escape Analysis (TEA) exercises from time to time for many Products (Hardware and Software). Some even have an integrated and automated mechanism in place to dig up Test Escape data usually based on ‘Externally Found Defects (EFD)’ or ‘Customer Found Defects (CFD)’ logged by the Support Escalation Team
• EFDs and CFDs should be carefully analyzed in context with Customer’s Live deployment from an end-to-end perspective, not just the Infrastructure but also the end user devices, applications, traffic patterns

Understanding Customer Verticals:

Customers usually fall into one of the below broader verticals:

• Healthcare
• Retail
• Finance
• Education
• Transportation
• Manufacturing
• Engineering
• Federal (Govt)

Activities

#1) Develop a separate Test Plan and Test Case for Longevity Testing. This will also help us track test execution, bug logging, and verification

#2) Identify test cases based on Test Escape Analysis inputs – usually bug scrub of EFDs or CFDs

#3) It is very important that QA team mimics test beds of one or more verticals depending on the organization’s line of business with number of verticals

#4) Dedicated Test Bed(s) should have

• Network Topology is similar to that of an intended vertical or multiple verticals
• Infrastructure has similar switches, routers, back-end servers, firewalls etc.
• Most frequently and popularly used application servers from a given vertical(s)
• Most frequently and popularly used end-user gadgets from a given vertical(s)

#5) Appropriate tools for generating Load, Stress and Real-Time Traffic

#6) Identify Manual execution resource

#7) Identify Automation resource/strategy for faster and repeater execution

#8) Identify START and END of Longevity Testing for a given release

Two approaches for START and END of Longevity Testing:

I) Approach 1:

• Software code and Hardware should be in a stable condition
• START at the end of FEATURE Test Completion
• END before Code Freeze

II) Approach 2:

• Take a minor hit by allowing slightly unstable code
• START at the 70% completion of FEATURE test cycle
• END before Code Freeze

#9) Bug verification for resolved defects

#10) Move Longevity Testing to Regression for subsequent Regression Testing

Execution

• Set-up the Testbed(s) to mimic one or more Customer Verticals
• Ensure that all back-end Infra, Applications and Databases including flavors are similar to that of the customer’s
• Ensure end-user devices similar to that of the customer’s use are available and used during Test Plan execution
• Ensure that appropriate tools are available to generate moderate Stress and Loading of the System or Product
• Execute the Entire Test suite from the Longevity test Plan without a soft or hard reboot of SUT, DUT, back-end servers, or other Infra related devices
• Multiple runs of tests should be run in the above fashion for a defined non-stop duration from the slot 72-720 hrs.
• Record the results
• Log all the bugs identified
• Verify all the bugs

Pros and Cons of Longevity Testing

Pros

• Helps identify critical bugs before the customer finds them
• Helps stabilize the system or product for its serviceable features that are critical to the customer’s productivity and business
• Helps increase Customer Satisfaction
• Saves lots of costs $$$ to the Organization’s – money saved is money earned!!!
• Longevity testing reports can also be turned into Quality Certification proof catering to different verticals

Cons

• Initial cost to include Longevity Testing and its related activities as part of a given release and Regression activities
• Ideally suited for Waterfall model
• Agile/Scrum models need tweaking of duration and coverage

Conclusion

Many of the ‘Defects’ that arise out of Customer reported issues are primarily due to Test Escape. This, in turn, begs a lot of questions like Test Plan development, review, coverage and execution.

Externally Found Defects (EFD) and Customer Found Defects (CFD) have a business ($$$) impact for the Customer as well as for the Product Organization.

Longevity Testing being unique should help any Product organization improve Customer Satisfaction by identifying and resolving defects before the customer catches them. Longevity Testing also helps improve stability resulting in robust quality Systems and Products.

About the author: This article is written by STH author Vinayak. He has 12 years of QA/testing experience in Fortune 500 companies.

Let us know if you have any questions or suggestions about this article.