A Complete Overview of Non-Functional Testing:
Testing is broadly classified into two types:
- Functional Testing
- Non Functional Testing
In this tutorial we will learn about the second one which is Non-Functional Testing. This tutorial will give you an in-depth knowledge on the concept of Non-functional testing in simple terms for your easy understanding.
What You Will Learn:
- What is Non-Functional Testing?
- Purpose of Non-Functional Testing
- Importance of Non-Functional Testing
- Advantages of Non-Functional Testing
- How to Capture Non-Functional Requirements
- Differences in Functional & Non-Functional Testing
- Differences in Functional & Non Functional Requirements
- Non-Functional Testing is a Black Box or White Box Testing
- Non-functional Testing Types
- Performance Testing:
- Load Testing:
- Stress Testing:
- Volume Testing:
- Usability Testing:
- User Interface Testing:
- Compatibility Testing:
- Recovery Testing:
- Instability Testing:
- Documentation Testing:
- Failover Testing:
- Security Testing:
- Scalability Testing:
- Compliance Testing:
- Endurance Testing:
- Localization Testing:
- Internationalization Testing:
- Reliability Testing:
- Portability Testing:
- Baseline Testing:
- Efficiency Testing:
- Disaster Recovery Testing:
- Maintainability Testing:
- Non-functional Testing Tools
What is Non-Functional Testing?
Non-functional testing is done to verify the non-functional requirement i.e. Performance, Usability etc. It verifies if the behavior of the system is as per the requirement or not. It covers all the aspects which are not covered in functional testing.
As the name itself depicts, this testing concentrates on the non-functional aspect of the application. So what are the non-functional aspects? Or should I say what are the features which are not related to the functionality of the application?
Well, here are the answers to those:
- How does the application perform under normal circumstances?
- How does the application behave when too many users log in concurrently?
- Can the application handle stress?
- How secure is the application?
- Can the application recover from any disaster?
- Can the application behave in the same way in a different environment or OS?
- How easy is to port the application in a different system?
- Are the documents/user manual provided with the application easy to understand?
The list can keep going. But the point here is, are not these features contributed towards the quality of the application? The answer is YES. These features are equally important.
Imagine that an application meets all the user requirements perfectly, but some unauthorized user easily goes and cracks the data entered by the user in the application, or the application dies when more than 5BB of any file is uploaded. So would you say that the application is of good quality? Obviously not right!!
Purpose of Non-Functional Testing
The sole purpose of non-functional testing is to ensure that the non-functional aspects of the application are tested and the application works well in context to the same.
The purpose is to cover the testing of all the characteristics of the application which helps to provide an application that meets the business expectation.
Importance of Non-Functional Testing
Non-functional testing is as important as functional testing.
Functional testing tests the application’s functionality and ensures that it works as expected but non- functional testing ensures that the application works good enough to meet the business expectations.
In order to understand its importance let’s see a simple Example:
An application is developed and is completely tested for functionality but non- functional testing is not performed on the same.
And when the application goes live it might result in critical or major issues like when the load is increased on the application it becomes too slow and takes a lot of time to open.
Response time might increase or when the load is increased to an extent, the application might crash. This shows how important it is to test an application’s non-functional aspects.
Advantages of Non-Functional Testing
Given below are few advantages of Non-Functional Testing:
- Non-functional testing covers the testing which cannot be covered in functional testing.
- It ensures that the application runs efficiently and is reliable enough.
- It ensures the security of the application.
How to Capture Non-Functional Requirements
While we perform testing, the focus is mainly on functional testing which tests the functionality of the product. But non-functional testing is as important as functional testing and its requirement should be taken into consideration right from the inception of the product.
In agile non-functional requirement should be captured using inputs.
A non-functional requirement should be captured as:
- User /Technical Stories
- In Acceptance criteria
- In Artifact
#1) User /Technical Stories:
A non-functional requirement can be captured using user stories or technical stories. Capturing Non-functional requirements as a user story is same as capturing any other requirement. The only difference in user and a technical story is that the user story requires discussion and has visibility.
#2) Acceptance Criteria:
Acceptance Criteria is the point defined for accepting the product by the customer i.e. to get the product accepted the defined points should be in pass state.
A non-functional requirement should be included in acceptance criteria but sometimes it’s not possible to test the non-functional requirements with every story i.e. with every iteration. So, the requirements should be added or tested with the relevant iteration only.
#3) In Artifacts:
A separate artifact should be prepared for the non-functional requirements which would help to have a better idea of what needs to be tested and how it can be done in iterations.
Differences in Functional & Non-Functional Testing
Given below are few differences between Functional and Non-Functional Testing:
|S.No||Functional Testing||Non Functional Testing|
|1||Functional Testing verifies how well the system works.||Non functional testing verifies how well the system responds.|
|2||Functional testing verifies the functionality of the system or how the system behaves.||Non functional testing verifies the Performance of the system.|
|3||Functional testing is based on the customer requirement.||Non functional testing is based on the customer expectation.|
|4||Functional testing has following types:|
• Unit testing
• Integration testing
• System Testing
• Acceptance Testing
|Non functional testing includes:|
• Performance testing
• Load Testing
• Stress testing
• Volume testing
• Security testing
• Installation testing
• Recovery testing
|5||Example of Function testing: Providing username and password on login screen should allow the user to log into application.||Example of Non Function testing: Providing username and password on login page should allow to log into site within 3 seconds.|
|6||Customer feedback helps in reducing risk factors of the product.||Customer feedback is more valuable for non functional testing as it helps to improve and let the tester know the expectation of the customer.|
|7||Functional testing does not necessarily require tools. It can be done effectively manually.||Non functional testing can be more appropriate/efficient using tools.|
|8||Functional testing is done based on business requirement.||Non- functional testing is done based on customer expectation and Performance requirement.|
Differences in Functional & Non Functional Requirements
There are several differences between the functional and non-functional requirements and few of them are stated below:
|S.No.||Functional Requirement||Non functional Requirement|
|1||Functional requirement is Customer based.||Non Functional Requirement is based on developers and technical knowledge of the team.|
|2||Functional Requirement specifies which functionality to be taken into consideration i.e. what needs to be tested.||Non functional requirements specify how it needs to be tested.|
|3||Functional testing is performed before the application goes live.||Non functional requirements includes the maintenance testing, Documentation testing which are not required while execution is going on but one the application has gone live.|
|4||It is known as functional requirement only.||Also known as Quality requirements.|
|5||The implementation plan for functional requirement is defined in system design document.||The implementation plan for non functional requirement is defined in system architecture.|
|6||Functional requirement includes testing of technical functionality of the system.||Non functional requirement includes qualities like security ,usability etc.|
Non-Functional Testing is a Black Box or White Box Testing
Non-functional testing comes under black box testing technique.
This technique is not limited to test the functionalities only but can be used for testing the non-functional requirements as well as Performance, usability etc. Black box testing technique does not require knowledge of the internal system i.e. does not require knowledge of code to the tester.
A checklist is used to ensure that no important aspect is left without testing.
A checklist is generally used when there is no time for documentation and the product has to be tested or when there is a time constraint, a checklist can be used to ensure that all the important aspects have been covered.
Let’s see an Example of Performance, Security & Documentation testing checklist.
Checklist for Performance Testing:
- Response time of the application should be verified i.e. how long does it take to load the application, any input given to the application provides the output in how much time, refreshing the browser etc.
- Throughput should be verified for the number of transactions completed during a load test.
- Environment set up should be the same as the live environment or else the results would not be the same.
- Process time – Process activities like import & export of excel, any calculations in the application should be tested.
- Interoperability should be verified i.e. a software should be able to inter-operate with the other software’s or systems.
- ETL time should be verified i.e. time taken in extracting, transforming and loading the data from one database to another.
- Increasing Load on the application should be verified.
Checklist for Security testing:
- Authentication: Only the authentic user should be able to Log in.
- Authorized: User should be able to log into those modules only for which he is authorized or for which the user has been provided access to.
- Password: Password requirement should be verified i.e. password should be as per how the requirement defines i.e. length, special characters, numbers etc.
- Timeout: If the application is inactive then it should timeout in a specified time.
- Data Backup: Data backup should be taken at a specified time and should be copied to a secured location.
- Internal links to the web application should not be accessible if placed directly in the browser.
- All the communication should be encrypted.
Checklist for Documentation Testing:
- User & System documentation.
- Documents for training purpose.
Non-functional Testing Types
The Following image depicts the types of Non-Functional Testing:
Evaluates the overall performance of the system.
Key elements are as follows:
– Validate that the system meets the expected response time.
– Evaluate that the significant elements of the application meet the desired response time.
– It can also be conducted as a pert of integration testing.
– It can also be conducted as a part of system testing.
Evaluates whether the system’s performance is as expected under normal and expected conditions.
Key points are:
– Validate that the system performs as expected when concurrent users access the application and gets the expected response time.
– This test is repeated with multiple users to get the response time and throughput.
– At the time of testing, the data base should be realistic.
– The test should be conducted on a dedicated server which stimulates the actual environment.
Evaluates whether the system’s performance is as expected when it is low on resources.
Key points are:
– Test on low memory or low disc space on clients / servers that reveals the defects which cannot be found under normal conditions.
– Multiple user perform the same transactions on the same data.
– Multiple clients connected to servers with different workloads.
– Reduce the Think Time to “Zero” to stress the servers to their maximum stress.
Think Time: Just like the time interval between typing your user and password.
Evaluates the behavior of the software when a large volume of data is involved.
Key points are:
– When the software is subject to large amounts of data, checks the limit where the software fails.
– Maximum database size is created and multiple client query the database or create larger report.
– Example– If the application is processing the database to create a report, a volume test would be to use a large result set and check that if the report is printed correctly.
Evaluates the system for human use or checks if it is fit for use.
Key points are:
– Is the output correct and meaningful and is it the same as which was expected as per the business?
– Are the errors diagnosed correctly?
– Is the GUI correct and consistent with the standard?
– Is the application easy for use?
User Interface Testing:
Evaluates the GUI.
Key points are:
– GUI should provide help and tool tips to make it easy for use.
– Consistent for its look?
– Data is traversed correctly from one page to another?
– GUI should not annoy the user or get difficult to understand.
Evaluates that the application is compatible with other hardware /software with minimum and maximum configuration.
Key points are:
– Test with each hardware with minimum and maximum configuration.
– Test with different browsers.
– Test cases are the same like which were executed during functional testing.
– In case the number of hardware and software are too many, we can use OATS techniques to arrive at the test cases to have maximum coverage.
Evaluates that the application terminates gracefully in case of any failure and the data is recovered appropriately from any hardware and software failures.
The tests are not limited to the below points:
– Power interruption, to the client while doing CURD activities.
– Invalid database-pointers and keys.
– Database process are aborted or prematurely terminated.
– Database pointers, fields and keys are corrupted manually and directly within the database.
– Physically disconnects the communication, power turn off, turn down the routers and network servers.
Evaluates and confirms if the software installs and uninstalls correctly.
Key points are:
– Validates that the system components are installed correctly on the designated hardware.
– Validates that the navigation on the new machine, updates the existing installation and older versions.
– Validates that with insufficient disc space, there is no unacceptable behavior.
Evaluates the documents and other user manuals.
Key points include:
- Validate that the stated documents are available in the product.
- Validates all the user guides, set up instruction, read me files, release notes and online help.
Failover testing is done in order to verify that in case of a system failure the system is capable enough to handle extra resources like servers. To prevent such a situation, backup testing plays a big role. Creating a backup system is what the process is about. If the backup is available, then it helps to get the system back.
Security testing is done to ensure that the application has no loopholes which could lead to any data loss or threats. Security testing is one of the important aspects of non-functional testing and if not performed properly can lead to security threats.
Security testing includes testing authentication, authorization, integrity, and availability.
Scalability testing is done to verify if the application is capable enough to handle the increased traffic, number of transactions, data volume etc. The system should work as expected when the volume of data or change in the size of data is done.
Compliance testing is done to verify if the standards defined are being followed or not. Audits are done to verify the same.
For Example, Audits are done to verify the process of creating test cases/test plans and placing them in the shared location with the standard name is being done or not. In QC while naming the test cases the standard test case name is being followed or not.
Documentation is complete and approved or not. These are the few points which are covered while auditing.
Endurance testing is done to verify the system’s behavior when a load is increased to an extent for a long time.
Endurance testing is also called as Soak testing & Capacity testing. It helps to verify if there are any memory leaks in the system. Endurance testing is a subset of load testing.
Localization testing is done to verify the application in different languages i.e. different locales. The application should be verified for a particular culture or locale. The main focus is to test the content, GUI of the application.
Internationalization testing is also known as i18n testing. I18n represents I –eighteen letters- N. Internationalization testing is done to verify if the application works as expected across all the language settings. It verifies that any functionality or application itself does not break i.e. the application should be capable enough to handle all the international settings.
Internationalization testing also verifies that the application gets installed without any issues.
Reliability testing is done to verify that the application is reliable and is tested for a specific period of time in the defined environment. An application should give the same output as expected every time only then an application can be considered as reliable.
Portability testing is done to verify that in case a software/ application is installed on a different system or on a different platform it should be able to run as expected i.e. no functionality should be affected because of a change in the environment.
While testing it is also required to test the change with the hardware configuration such as the hard disk space, Processor and also with different operating systems to ensure that the applications correct behavior and expected functionality are intact.
Baseline testing is also known as benchmark testing as it creates a base for any new application to be tested.
For Example: In the first iteration, the response time for an application was 3 seconds. Now, this has been set as a benchmark for the next iteration and in the next iteration, the response time changes to 2 seconds. Baseline testing is basically a validation document which is used as a base for future references.
Efficiency testing is done to verify if the application works efficiently and the number of resources required, tools required, complexity, customer requirement, environment required, time, what kind of project it is are few points which would help to define how efficiently an application would work if all the considered parameters work as expected.
Disaster Recovery Testing:
Disaster recovery testing is done to verify the success rate of recovery of the application or system if any critical failure happens and weather the system is able to restore the data and application or the system could cope up easily to return the way it was working earlier i.e. from the operational front.
Once the application/Product goes live, there are chances of that issue coming up in the live environment or the customer wants enhancement for the application which is already live.
In this case, maintenance testing team is available to test for the above scenarios mentioned. Once the application goes live it still needs maintenance for which the maintenance testing team works.
Non-functional Testing Tools
There are several tools available in the market for Performance (Load & Stress) testing.
Few of them are listed below:
- Load Complete
- Webserver Stress Tool
- WebLoad Professional
Non-functional testing is one of the important parts of Software testing and testing cannot be considered complete without this as it ensures that application/Product behavior is acceptable and is as per the business need.
Non-functional testing also requires a strategy and planning.
We can include non-functional testing details in the test plan or can create a separate strategy for it. In either case, the goal is to have a proper coverage of non-functional aspects of the software.
Hope you would have immense knowledge in Non-Functional Testing concept.