How to Perform Software Product Testing – Detailed Process and Methods with Examples

Software Products need their own unique approach to test adequately and correctly. Often times, teams treat them as any other software (i.e. internal applications built for a specific client or team; not accessible by the general public; non-revenue generating) and that is the starting point of trouble.

Software Product Testing needs a custom test style and strategy to add value. Software Product development and sustenance is in itself a complex ecosystem and to thrive testers need to adapt.

Let me take a moment to explain why it is important and why I think Product development is complex, complicated and composite, even at the best of times.

Software Product development challenges:

Here are some of the challenges that Software Product development teams face:

#1) Lack of control over user demographics, devices, environments, platforms, etc.: Software products unlike software built for specific stakeholders are not used in controlled and predictable situations. There are many just too many factors to take into account.

#2) Foggy product vision: Product behavior and features are forever changing and the journey to maturity isn’t clearly visible. Or the product is growing too rapidly that it spirals out of control that teams don’t know what is happening.

#3) Aggressive timelines: Due to heavy competition in the software product market, things have to move at a breakneck speed and teams must stay a step ahead of their peers. Otherwise, they are sure to lose out to the competition.

#4) Fear of failure: Software products are usually innovative. So, their success is not always a given. This is the reason companies can’t go all out in terms of budget, technologies, infrastructure, etc. They often have to hold back to gain a certain amount of immunity from failure or to even breakeven.

#5) Lack of actionable feedback: Since there are no stakeholders or business users or clients, so to say, it is difficult to understand what the end user may or may not like. Companies are constantly playing a guessing game and often have difficulty in bridging the gap between what they want for the software and what the customer wants.

These challenges affect all areas of product development, marketing, and sustenance- And they inherently impact product testing too.

To get ahead in the game, this type of testing has to take five key points into account:

• Speed of development and releases
• Short term and Long term product goals of the product
• Extent and nature of competition
• Target audiences and their environments
• Requirements – Functional, performance, security, usability, configuration, etc.

Before we go into more details, let’s understand product life cycle (This is a generic product lifecycle and not specific to software products but software follows a similar pattern):

A good Product Test Strategy/Approach should take into consideration the current stage of the product in its life cycle.

Also read => How to write a good test strategy document

Example: A company XYZ’s product is a defect tracking software called ‘TrackFast’.  It is a new product and the first version is set to be launched as a cloud and on-premise solution. TrackFast works like any other defect management system and is built for both Mobile and Web access. Currently, there are 2 to 4-week sprints at which the product is created in parts. You are on the testing team testing ‘TrackFast’ before it meets its customers. The testing involves checking functionality, performance, and security.

To summarize, these are the parameters you are working with. Or if you prefer, this is your context

Let’s see how to test at each stage. This is product test process, method, or life-cycle at each stage.

Stage #1) Product Introduction

Since this is the first time TrackFast would be going out into the market, the idea is to make a good first impression. So leave no stone unturned. Test everything and from every angle. In addition to that, lay the foundation for future testing.

A good test strategy at this point should include the following:

• Tests that validate the short term goals of TrackFast. “What does it need to be shipped correctly” should be at the forefront of the testing effort. Create End to end tests (front end, middleware, and backend) for thorough testing of every feature
• Tests that compare TrackFast with the competition (ideally this is the job of product owners but as a tester we can add our two cents. Also, this step is easier if the software has some peers already. For example: It is easy to compare TrackFast with Bugzilla or JIRA or other legacy systems. But let us say I am creating an app that does something unusual like being able to predict when a baby is hungry or cranky :), it might be hard to find an application that you can use as a baseline)
• Platform, browser and device compatibility tests
• Tests for ease of installation, set up and getting-up-to-speed
• Tests for performance, security, and usability
• Integration Tests if it interfaces with other systems. A simple integration example is that Defect tracking systems often interact with email clients to send notifications
• Plan for regression– It is a good idea to flag or mark critical tests that you think will be a part of future regression cycles and think about automating them for future releases
• Plan for known issues (are you going to be adding them to the backlog or handling them as CRs, etc.)
• Flexibility to change when the product progresses to the next life cycle stage.

It could sometimes be a long wait before the product goes out, so use all the time you have to do as thorough a job as possible.

In this stage, though there is a piece of the product ready at the end of 2-4 week sprints, most often every sprint does not result in shipped code. Therefore, never consider last sprint testing ‘done-and-delivered’. Repeat critical tests with every sprint until release. With each sprint, test the entire product that you have until that point.

Stage #2) Product Growth

After the initial project introduction, if all goes well, expect an influx of activity because Product Growth is a fast paced lane. You are now swimming along with the big sharks and unless you keep up, you get gobbled down.

Here, the releases get shorter, the improvements done to the software become more in number and extent of regression almost becomes unmanageable.

The product testing strategy should work with the pace that the software development is proceeding and should not become a bottleneck.

These can help:

• Keep in mind the long-term goals of the project. It is not about getting-it-over-with now. It is about living with the features and thriving with them.
• Test Early- Consider TDD or BDD instead of deferring testing to the end with new requirements
• Automate Regression and strengthen it– Create an automated regression suite in place so you are not left with untested landmines in your system
• If your business/product owners want to get involved with testing, consider a business language based automation tool such as Cucumber.
• Keep usability and site design central to your testing. Because the more features we add, the cleaner the site should look
• Perform performance and security testing when a major release has happened or there is a significant change made to the architecture. (New server brought in, etc.) Most software systems don’t need this with every release.
• Keep in touch with the competition and know the product vision
• Adapt pair testing, for immediate feedback and fixing. Include the product owner when possible
• Plan for changes and known issues
• Try to get your hands on the customer feedback and check if they are can be tracked as enhancement suggestion to keep the growth constant. (once again, this is not the primary responsibility of the QA team, but everyone counts)

Stage #3) Product Maturity

Congratulations that your product has come this far. At this point, the features don’t change as often. The product team is going to be more focused on bringing more business or their marketing efforts. However, product development and testing need not and often don’t stop.

Therefore, the testing team can:

• Work on maturing your test strategy. By this point, your regression suites, test design methods, and test management practices must work like well-oiled machines.
• Focus on the finer details. Because overall the product works and is doing well, but as they say- ‘God is in the details’– find even the smallest of the problems that can improve the quality of the system
• Consider customer feedback
• Test Performance and security periodically
• Take into account the new devices, platforms, and browsers that might have come into the market from the last time you tested
• Test User Manual and FAQ pages because by now you have the time and you can afford to.
• Experiment with a new product test tools, services or a process because now you can.
• Test the installation process with every release, however small that might be and get statistics as to how easy or difficult it is for the end user.

Whatever you do, don’t get complacent.

Stage #4) Product Decline/Circling back to Product growth

The product owners and businesses are smart these days and know very well that they can’t keep their product the same and expect the users to stay loyal. Things move too fast and so do products.

So, TrackFast can’t sit back and relax. If it needs to have a continued market presence and stay the leader, it needs to evolve. Like it or hate it, Facebook started as a simple social network to connect people and it is a large software platform in itself integrating with a million other things and staying current.

TrackFast too has to evolve. After proving that it is a reliable and effective defect tracking system, it has to evolve or it will decline. So, the company XYZ decides to improve TrackFast by making it a general ticketing system that can be used to track any incidents or cases by the business other than IT/test teams (something like JIRA) and not just for defects in the software development process.

The wheel has made a full turn and you find yourself treating the system as a brand new one and follow the strategy we discussed in the Product Introduction section. Only now you are more experienced and familiar with the drill. But remember, with each new turn comes a new challenge. So stay sharp :)

What makes you a successful product tester?

• Product testers must have a keen business sense, understanding of fast delivery development models and should be ace testers who are not afraid to experiment with tools and become a bit of coders themselves if need be. These things can have a positive impact on any type of testing, but they are an absolute necessity in this type of testing.
• Another important quality is that a product tester must believe in the product and genuinely want it to succeed. When I as a tester think that the software is total garbage, there is little hope that I will do anything to make it better.
• Share the product/business owner’s vision. Unless you know where the product is going and how it is going to evolve, the testing will be super limited.
• Cross-functional skills are beneficial– Know how to test the DB, how to take performance benchmarks, how to enable security certificates, how to deploy, etc. Be curious and explore.
• Set no boundaries– don’t think that evaluating the user manual or checking the FAQs is not your job and a technical writer should take care of it. Well, they should and they will. But when you look at it as an insider as someone who knows the product inside out, your feedback is super useful.
• Seek end user feedback. The next big set of people who test after you are the real time users. Know and understand what kind of problems they are facing. This helps you improve your test design so next time you know what to do to avoid them those problems.
• Work fast and be a decision maker
• Avoid technical debt. In a fast development and testing situation, it is easy to test exploratorily exclusively and loose the frame of reference for the future releases. Don’t let this happen. Maintain skeletal documentation so you can track, trace and measure

The biggest difference between testing software built as a service and software built as a product is that – in the former, once the test strategy is arrived at, it is applied for all subsequent testing.

However, for a product the test strategy has to change depending on the current life cycle stage the product is in and the changes to the market dynamics (new devices, new browsers, etc.). Product testing strategy needs to be much more flexible to change.

About the author: This article is published by STH team member Swati S.

We hope this article has been useful. Please feel free to post your comments, questions, and feedback below.