So…do you really think it will take that huge amount of time to test, what you call system testing, even after spending lot of efforts on integration testing?
The client we recently approached for the project was not convinced about the estimation we provided for each testing effort.
I had to chime in with an example –
Mike, I would like to elaborate our efforts and importance of system testing with an example.
Shoot, he replied.
A car manufacturer does not produce car as a whole car. Each component of car is manufactured separately, like seats, steering, mirror, break, cable, engine, car frame, wheels etc.
After manufacturing each item, it is tested independently whether it is working the way it is supposed to work and that is called Unit testing.
Now, when each part is assembled with other part, that assembled combination is checked if assembling has not produced any side effect to the functionality of each component and whether both components are working together as expected and that is called integration testing.
Once all the parts are assembled and car is ready, it is not ready actually.
The whole car needs to be checked for different aspects as per the requirements defined like if car can be driven smoothly, breaks, gears and other functionality working properly, car does not show any sign of tiredness after being driven for 2500 miles continuously, color of car is generally accepted and liked, car can be driven on any kind of roads like smooth and rough, sloppy and straight etc and this whole effort of testing is called System testing and it has nothing to do with integration testing.
The example worked the way it was expected and client was convinced about efforts required for ST (system testing).
I narrated the example here to encourage importance of ST.
What You Will Learn:
So, what is System Testing?
Definition: As per IEEE, it is – The process of testing of an integrated hardware and software system to verify that the system meets its specified requirements.
Verification: Confirmation by examination and provisions of objective evidence that specified requirements have been fulfilled.
It is performed when integration testing is completed.
It is mainly a black box type testing. This testing evaluates working of system from user point of view, with the help of specification document. It does not require any internal knowledge of system like design or structure of code.
It contains functional and non-functional areas of application/product.
It mainly focuses on following:
- External interfaces
- Multiprogram and complex functionalities
- Operator and user’s smooth interaction with system
- Load / Stress
Why System Testing?
#1. It is very important to complete a full test cycle and ST is the stage where it is done.
#2. ST is performed in environment which is similar to the production environment and hence stakeholders can get a good idea of the user’s reaction.
#3. It helps to minimize after-deployment troubleshooting and support calls.
#4. In this STLC stage Application Architecture and Business requirements both are tested.
How to perform System test?
It is basically a part of software testing and test plan should always contain specific space for this testing.
To test the system as a whole, requirements and expectations should be clear and the tester needs to understand real time usage of application too.
Also, most used third party tools, version of OSes, flavors and architecture of OSes can affect system’s functionality, performance, security, recoverability or installability.
Therefore, while testing system a clear picture about how the application is going to be used and what kind of issues it can face in real time can be helpful. In addition to that, requirements document is as important as understanding the application.
Clear and updated requirements document can save tester from number of misunderstandings, assumptions and questions.
In short, a pointed and crisp requirement document with latest updates along with understanding of real time application usage can make ST more fruitful.
Example Test Scenarios
System testing sample test scenarios for an eCommerce Site:
1. If the site launches properly with all the relevant pages, features and logo
2. If the user can register/login to the site
3. If the user can see products available, can add products to his cart can do payment and can get confirmation via e-mail or SMS or call.
4. If the major functionality like searching, filtering, sorting, adding, changing, wish list etc work as expected
4. If number of users (defined as in requirement document) can access the site simultaneously
5. If the site launches properly in all major browsers and their latest versions
6. If the transactions are being done on the site via specific user are secure enough
7. If the site launches properly on all the supported platforms like Windows, Linux, Mobile etc.
9. If the content of pages are properly aligned, well managed and without spelling mistakes.
10. If session timeout is implemented and working as expected
11. If user is satisfied after using the site or in other words user does not find it difficult to use the site.
Types of System Test:
ST is called a super set of all types of testing as all the major types of testing are covered in it. Although focus on types of testing may vary on the basis of product, organization processes, timeline and requirements.
The overall it can be defined as below: (click on image to enlarge)
Installation Testing: To make sure that product / software can be installed on specific or support defined system, can be configured and can be brought into an operational mode.
Functionality Testing: To make sure that functionality of product are working as per the requirements defined, within the capabilities of the system,
Recoverability Testing: To make sure how well the system recovers from various input errors and other failure situations.
Interoperability Testing: To make sure whether the system can operate well with third party products or not.
Performance Testing: To make sure system’s performance under various condition, in terms of performance characteristics.
Scalability Testing: To make sure system’s scaling abilities in various terms like user scaling, geographic scaling and resource scaling.
Reliability Testing: To make sure system can be operated for longer duration without developing failures.
Regression Testing: To make sure system’s stability as it passes through integration of different sub systems and maintenance tasks.
Documentation Testing: To make sure that system’s user guide and other help topics documents are correct and usable.
Security Testing: To make sure that system does not allow unauthorized access to data and resources.
Usability Testing: To make sure that system is easy to use, learn and operate.
Tips for fruitful System testing:
Tip #1: Replicate real time scenarios rather than doing ideal testing as the system is going to be used by an end user and not by the trained tester.
Tip #2: Verify system’s response in various terms as human does not like to wait or to see wrong data.
Tip #3: Install and configure system as per the documentation because that is what end user is going to do.
Tip #4: Involving people from different areas like business analysts, developers, testers, customers can end in a better system.
Tip #5: Regular testing is the only way to make sure that littlest change in code to fix the bug has not inserted another critical bug into the system.
- Software development -> completed
- All modules are integrated fully and integration is tested
- Unit testing -> completed
- Production like test environment is ready
What is System Integration Testing?
It is a type of testing in which system’s ability to maintain data integrity and operation in co-ordination with other systems in same environment, is being checked.
Example of System Integration Testing:
Let’s take an example of well-known online ticket booking site – http://irctc.co.in.
This is a ticket booking facility; online shopping facility interacts with PayPal. Overall you can consider it as A*B*C=R.
Now on system level, online ticket booking facility, online shopping facility and online payment option facility can be system tested independently, followed by check perform Integration tests for each of them. And then the whole system needs to be system tested.
So where does System Integration testing come into the picture?
The web portal http://Irctc.co.in is a combination of systems. You may perform tests at the same level (single system, system of systems), but at each level you may want to focus on different risks (integration problems, independent functionality).
- While testing Online Ticket booking facility, you may verify if you are able to book tickets online. You may also consider integration problems e.g. Ticket booking facility integrates back-end with front-end (UI). For example, how front-end behaves when database server is slow to respond?
- Testing of Online ticket booking facility with online shopping facility. You may verify that online shopping facility is available for the users logged into the system to book tickets online. You may also consider verification of integration in online shopping facility. For example, if user is able to select and buy a product without hassle.
- Testing of online ticket booking facility’s integration with PayPal. You may verify whether after booking tickets, money were transferred from your PayPal account to Online Ticket Booking account. You may also consider verification of integration in PayPal. For example, what if system puts two entries in database after debiting money for once only?
Difference between System Testing and System Integration Testing:
The main difference is –
- System Testing looks after single system’s integrity with relevant environment
- System Integration Testing looks after multiple systems’ integrity with each other, being in same environment.
Thus, ST is the beginning of real testing where you test a product as a whole and not a module / feature.
With this, we are ending our System Testing session and again ready to discuss with you readers. As usual, please pour in your comments and views about the post and we would like to connect with our readers to make the software testing community stronger.
About the author: This guide is written by STH team member Bhumika Mehta. She is a project lead, carrying more than 7 years of experience in software testing.