Introduction to Software Configuration Testing:
This article has been brought to you to discuss a special type of software testing which is very important for the business.
We will discuss the basics of configuration testing, its need, importance, goals, and types. As we will progress further in the article, we will learn in detail how to perform this testing.
Let us begin!!
This tutorial will give you a complete overview of Configuration Testing and it will act as a perfect guide to anyone who is new to this testing concept and brush up the knowledge of those who already have experience in it.
What You Will Learn:
- What is Configuration Testing?
- Why This Testing?
- Preparing for Configuration test
- Microsoft VSTS – Configuration Testing Tool
- Types of Configuration Testing
- Importance of this Testing for an Enterprise
- Recommended Reading
What is Configuration Testing?
Configuration testing is a method of testing a system under development on multiple machines that have different combinations or configurations of hardware and software. The performance of the system or an application is tested against each of the supported hardware & software configurations.
When we say different configurations of hardware & software, it is attributed to multiple operating system versions, browsers, supported drivers, memory sizes, hard drive types, CPU's, etc.
Why This Testing?
As discussed above, this testing is a technique that assesses the performance of a software, system/application considering the different system configurations.
This testing is done to determine the optimal configurations under which a system or an application can work fine without any bugs, issues or flaws in performance. So, the most efficacious configuration that will deliver the required performance characteristics is spotted with the help of this testing.
The second main reason for this testing is to verify the system’s compatibility with the other software or equipment signified in the SRS (software requirement specification).
Let's consider that your enterprise has developed a desktop application in C# language and this app is built on the .NET framework.
And this app is based on a 3-tier architecture which has three layers – front end (client), the application layer (server) and the database layer. Each of the layers will support certain platforms accordingly.
Let’s assume that each layer supports the below platforms:
Client – Windows 10 OS, Windows 7, Windows XP, Linux OS.
Database – Microsoft SQL Server, IBM DB2, MySQL.
Now, as a software tester, you will have to test the application under each of the various combinations of the above client, server and database platforms to ensure that the application works well with each of the possible configurations.
For Example, you will check how the application works with the combination of Windows 10 OS, Windows Server 2016 & MySQL database, then you will perform another test to check how the application works with the combination of Windows 10 OS, Windows Server 2016 and IBM DB2 database.
And, so on until you test all the possible configurations.
Our testing would not only be restricted to the software, but it will also cover the hardware, where we will have to check each of the combinations of various hardware devices. Therefore, at times this testing is also referred as Hardware Configuration Testing.
Preparing for Configuration test
This testing has certain pre-requisites that need to be fulfilled before we execute the configuration tests.
Below are the prerequisites:
#1) Preparing the Coverage Matrix
Owing to a huge number of possible hardware & software configurations, it becomes very time-consuming and almost impossible to test each of the configurations effectively.
For Instance, in the example that we discussed above, we will have a total of 3*3*3 i.e. 27 software configurations. And, let’s suppose we have 5 different hard drives and 6 different memory sizes. Then, the count will go to 27*5*9 i.e. 810 configurations now.
This will keep on increasing if we add more components to the picture. So, it becomes crucial to do the planning for the software testing effort and identify clearly that which platforms will be supported.
Then, we need to come up with a coverage matrix that will hold the various combinations of the hardware & software configurations. Sometimes, this coverage matrix is also known as BCM (Basic Configuration Matrix).
The above figure shows a sample schematic matrix of configurations that you would wish to test.
#2) Prioritizing the Configurations
Once the configuration matrix is prepared, the next step is to prioritize the configurations.
This step is required because it is impossible to test the whole huge wide range of configurations. So, based on the client feedback, the most critical configurations are enlisted, and they are to be thoroughly tested first.
Once, we are done with the above two steps, we can go ahead with testing the various configurations based on their priority.
Microsoft VSTS – Configuration Testing Tool
Microsoft Visual Studio Team Services (VSTS) is a tool which greatly helps in testing your app under various configurations based on your test plan.
You should have a test plan to decide which tests you want to execute and on which configurations. You need to ensure that you have the right environment set up for the configurations you require. Once you have the matrix of the combinations you need to test it.
You can follow the below steps to perform this testing:
#1) Set up the configurations and create the variables. A variable is one of the components in your configuration.
For Example, there can be a variable ‘Browser’ which can have multiple values like Chrome, Firefox, IE10, etc.
#2) Assign the configurations to the test plans/test suites or individual test cases.
#3) Execute the tests against each configuration.
#4) Track the test results for each of the configurations.
A detailed step by step guide to perform this testing with VSTS can be viewed here: VSTS – Test different configurations
Given below are the various goals of this Testing:
- To determine the optimal configuration of the AUT (Application under Test).
- To discover the hidden vulnerabilities contained by the hardware.
- To ensure the business continuity by detecting and fixing the issues before they hamper the business operations.
- To eliminate a configuration drift.
- To minimalize the number of failures due to host configuration errors.
- Validating the application to ascertain that it satisfies the configurability.
- To analyze the system performance by adding, removing or modifying the hardware components (E.g. Varying the memory sizes, adding load balancers, etc.).
- To verify if the system works fine in a geographically distributed environment (servers & clients are situated at different locations).
- Verifying how easily the bugs can be replicated regardless of the configuration changes.
- To verify how traceable and manageable the application items are.
Types of Configuration Testing
We can have two different lines of divisions for this testing.
On the first line, it can be divided into two parts:
- Software configuration Testing
- Hardware configuration testing
On the second line, it can again be divided into two parts – 1. Client level testing & 2. Server level Testing
To start with let's see about hardware and software configuration.
#1) Software Configuration Testing:
This testing is done over the AUT (application under test) with multiple OS versions, software updates, etc. This is a long-winded testing as it consumes a huge time to install and uninstall the various software which are to be used for testing.
One approach to save the time is to employ virtual machines for testing the software configuration. A VM simulates the real-time configurations and gives the same feel as of a physical machine.
So, instead of installing and uninstalling the multiple software on a single physical machine, we can have multiple virtual machines simulating each different configuration against which we need to do the testing.
Software configuration testing begins once the build is released after passing through the unit test and integration test.
#2) Hardware Configuration Testing:
Hardware configuration testing is usually carried out in labs, where we have physical machines with various hardware connected to them.
Every time a build is released, the software is required to be installed in all the physical machines to which the hardware is attached, and the test suite is required to be executed on each and every machine to confirm that the application is working fine.
To accomplish the above-stated task, a significant amount of effort is required to set up the software on each machine, connect the hardware to it and then manually execute the test suites. Even if we automate this task and run the test suite, even then a significant effort would be required.
Also, as we have already discussed in the former part of this article, whilst doing hardware configuration test, we spell out the kind of hardware to be tested, and there are numerous computer hardware and peripherals which make it next to impossible to execute all the tests.
So, a tester analyzes that which hardware is mainly used by the customer and then conducts the testing based on the prioritization.
Now let's discuss client level and server level configuration testing.
#1) Client level testing: It has a commonality with usability and functionality testing. This testing is done from the point of view of its direct user’s interests.
#2) Server level testing: This type of testing is done to investigate the interaction between the software and the external environment when it is planned to be integrated post-release.
Importance of this Testing for an Enterprise
This testing is very important for an enterprise to come up and sustain with high performing, scalable and available applications.
This testing should be performed in a continuing and preferably an automated manner (otherwise it is very time consuming when done manually) with the intention of quickly identifying and correcting the factors that contribute to inefficiencies.
Configuration testing is as important as any other software testing type as it discovers the optimal configuration of the system under which the application performs at its best. Also, it helps in identifying and rectifying any compatibility issues.
In this article, we learned about what is configuration testing, why do we perform this testing, what are its goals, what are its types, what are the prerequisites of this testing, how it needs to be done, how important is it for an enterprise, etc., in a detailed manner.
Hope, this article was informative and would have brought you a fair idea about configuration testing.