So…do you really think it will take that huge amount of time to test, what you call system testing, even after spending a lot of efforts on integration testing?
The client we recently approached for the project was not convinced about the estimation we provided for each testing effort.
I had to chime in with an example –
Mike, I would like to elaborate our efforts and importance of system testing with an example.
Shoot, he replied.
A car manufacturer does not produce the car as a whole car. Each component of the car is manufactured separately, like seats, steering, mirror, break, cable, engine, car frame, wheels etc.
After manufacturing each item, it is tested independently whether it is working the way it is supposed to work and that is called Unit testing.
Now, when each part is assembled with another part, that assembled combination is checked if assembling has not produced any side effect to the functionality of each component and whether both components are working together as expected and that is called integration testing.
Once all the parts are assembled and the car is ready, it is not ready actually.
The whole car needs to be checked for different aspects as per the requirements defined like if car can be driven smoothly, breaks, gears and other functionality working properly, car does not show any sign of tiredness after being driven for 2500 miles continuously, color of car is generally accepted and liked, car can be driven on any kind of roads like smooth and rough, sloppy and straight etc and this whole effort of testing is called System testing and it has nothing to do with integration testing.
The example worked the way it was expected and the client was convinced about efforts required for ST (system testing).
I narrated the example here to encourage the importance of ST.
What You Will Learn:
Definition: As per IEEE, it is – The process of testing of an integrated hardware and software system to verify that the system meets its specified requirements.
Verification: Confirmation by examination and provisions of objective evidence that specified requirements have been fulfilled.
It is performed when integration testing is completed.
It is mainly a black box type testing. This testing evaluates working of the system from a user point of view, with the help of specification document. It does not require any internal knowledge of system like design or structure of the code.
It contains functional and non-functional areas of application/product.
It mainly focuses on following:
Why System Testing?
#1. It is very important to complete a full test cycle and ST is the stage where it is done.
#2. ST is performed in an environment which is similar to the production environment and hence stakeholders can get a good idea of the user’s reaction.
#3. It helps to minimize after-deployment troubleshooting and support calls.
#4. In this STLC stage Application Architecture and Business requirements, both are tested.
It is basically a part of software testing and test plan should always contain specific space for this testing.
To test the system as a whole, requirements and expectations should be clear and the tester needs to understand the real-time usage of application too.
Also, most used third-party tools, version of OSes, flavours and architecture of OSes can affect system’s functionality, performance, security, recoverability or installability.
Therefore, while testing system a clear picture of how the application is going to be used and what kind of issues it can face in real time can be helpful. In addition to that, a requirements document is as important as understanding the application.
Clear and updated requirements document can save tester from a number of misunderstandings, assumptions and questions.
In short, a pointed and crisp requirement document with latest updates along with an understanding of real-time application usage can make ST more fruitful.
System testing sample test scenarios for an eCommerce Site:
1. If the site launches properly with all the relevant pages, features and logo
2. If the user can register/login to the site
3. If the user can see products available, can add products to his cart can do payment and can get confirmation via e-mail or SMS or call.
4. If the major functionality like searching, filtering, sorting, adding, changing, wishlist etc work as expected
4. If number of users (defined as in requirement document) can access the site simultaneously
5. If the site launches properly in all major browsers and their latest versions
6. If the transactions are being done on the site via specific user are secure enough
7. If the site launches properly on all the supported platforms like Windows, Linux, Mobile etc.
9. If the content of pages is properly aligned, well managed and without spelling mistakes.
10. If session timeout is implemented and working as expected
11. If a user is satisfied after using the site or in other words user does not find it difficult to use the site.
ST is called a superset of all types of testing as all the major types of testing are covered in it. Although a focus on types of testing may vary on the basis of product, organization processes, timeline and requirements.
The overall it can be defined as below: (click on image to enlarge)
Installation Testing: To make sure that product/software can be installed on specific or support defined system, can be configured and can be brought into an operational mode.
Functionality Testing: To make sure that functionality of product is working as per the requirements defined, within the capabilities of the system,
Recoverability Testing: To make sure how well the system recovers from various input errors and other failure situations.
Interoperability Testing: To make sure whether the system can operate well with third-party products or not.
Performance Testing: To make sure system’s performance under the various condition, in terms of performance characteristics.
Scalability Testing: To make sure system’s scaling abilities in various terms like user scaling, geographic scaling and resource scaling.
Reliability Testing: To make sure system can be operated for longer duration without developing failures.
Regression Testing: To make sure system’s stability as it passes through an integration of different subsystems and maintenance tasks.
Documentation Testing: To make sure that system’s user guide and other help topics documents are correct and usable.
Security Testing: To make sure that system does not allow unauthorized access to data and resources.
Usability Testing: To make sure that system is easy to use, learn and operate.
Tip #1: Replicate real-time scenarios rather than doing ideal testing as the system is going to be used by an end user and not by the trained tester.
Tip #2: Verify system’s response in various terms as the human does not like to wait or to see wrong data.
Tip #3: Install and configure the system as per the documentation because that is what end user is going to do.
Tip #4: Involving people from different areas like business analysts, developers, testers, customers can end in a better system.
Tip #5: Regular testing is the only way to make sure that littlest change in the code to fix the bug has not inserted another critical bug into the system.
It is a type of testing in which system’s ability to maintain data integrity and operation in coordination with other systems in the same environment, is being checked.
Example of System Integration Testing:
Let’s take an example of well-known online ticket booking site – http://irctc.co.in.
This is a ticket booking facility; online shopping facility interacts with PayPal. Overall you can consider it as A*B*C=R.
Now on the system level, online ticket booking facility, online shopping facility and online payment option facility can be system tested independently, followed by check perform Integration tests for each of them. And then the whole system needs to be system tested.
So where does System Integration testing come into the picture?
The web portal http://Irctc.co.in is a combination of systems. You may perform tests at the same level (single system, the system of systems), but at each level, you may want to focus on different risks (integration problems, independent functionality).
Difference between System Testing and System Integration Testing:
The main difference is –
Thus, ST is the beginning of real testing where you test a product as a whole and not a module/feature.
With this, we are ending our System Testing session and again ready to discuss with your readers. As usual, please pour in your comments and views on the post and we would like to connect with our readers to make the software testing community stronger.
About the author: This guide is written by STH team member Bhumika Mehta. She is a project lead, carrying more than 7 years of experience in software testing.