Application Testing is an activity that every software tester performs daily in his career. These two words are extremely broad in practical aspect. However, only the core and most important areas will be discussed here. The purpose of this article is to touch all the primary areas so that the readers will get all the basic briefing at a single place.

Categories of Applications

Whether it is small calculator software with only the basic arithmetic operations, or an online enterprise solution; there are two categories of applications.
a. Desktop
b. Web

For desktop applications, testing should take into account the UI, business logic, database, reports, roles and rights, integrity, usability and data flow. For web applications, along with all these major areas; testers should give sufficient importance to performance, load and security of the application. So AUT is either desktop software or a website.

Application Testing Methodologies

This is a well-known and well discussed aspect; there are only 3 universally accepted methodologies;

a. Black Box: In black-box testing, the AUT is validated against its requirements considering the inputs and expected outputs, regardless of how the inputs are transformed into outputs. Testers are least concerned with internal structure or code that implements the business logic of the application. There are four primary techniques to design test cases for black box testing;

i. BVA (Boundary value Analysis)
ii. EP (Equivalence Partitioning)
iii. Decision Tables
iv. State Transition Tables (and diagrams)

a. White Box: Primary focus of this methodology is to validate, how the business logic of application is implemented by code. Internal structure of the application is tested and the techniques available to do so are;

i. Code Coverage
ii. Path Coverage

Both the above listed techniques contain several other strategies that may be discussed in some other article. Some techniques are discussed in ‘Test Case Design Techniques’ topic.

a. Grey Box: Practically speaking, this is a mixture of black box and white box. In this methodology, mainly the tester tests the application as in black box. But, for some business critical or vulnerable modules of application; testing is done as white box.

Application Testing Tools

According to the best of my knowledge, there are at least 50 testing tools available in market today. These include both paid and open source tools. Moreover, some tools are purpose specific e.g. UI testing, Functional Testing, DB Testing, Load Testing, Performance, Security Testing and Link validation testing etc. However, some tools are strong and provide the facility of testing several major aspects of an application. The general concept of ‘Application Testing’ is its functional testing. So, our focus will be on functional testing tools.

Here is the list of some most important and fundamental features that are provided by almost all of the ‘Functional Testing’ tools.

a. Record and Play
b. Parametrize the Values
c. Script Editor
d. Run (the test or script, with debug and update modes)
c. Report of Run session

Different vendors provide some specific features that make their product unique to other competitor products. But the five features listed above are the most common and can be found in almost all the functional testing tools.

Following is the list of few widely used Functional Testing tools.

1) HP QTP (Quick Test Professional)
2) Selenium
3) IBM Rational Robot
4) Test Complete
5) Push to Test
6) Telerik

Software Test Plan (STP)

For any activity, some planning is always required and same is true for software testing. Without proper plan there is always high risk of getting distracted during the testing. If this risk becomes a fact, the results could be horrible.

Following are the 5 main parts of a good Test Plan:
a. Scope
i. Overview of AUT
ii. Features (or areas) to be tested
iii. Exclusions (features or areas not to be tested) with reason
iv. Dependencies (of testing activities on each other, if any)

b. Objectives: This section describes the goals of testing activity e.g. validation of bug fixes, new features added or revamp of AUT etc.

c. Focus: This section describes what aspect of application will be included in the testing e.g. security, functionality, usability, reliability, performance or efficiency etc.

d. Approach: This section describes what testing methodology will be adopted for which areas of AUT. For example, in the STP of an ERP application; the approach section may contain the information that black box testing will be approach for payroll. On the other hand, for reports the approach will be grey box testing.

e. Schedule: This sections describes that who will be doing what and where on the AUT, when and how. Schedule section is, in fact, a ’4Ws and H’ of the STP. Normally it is a simple table, but every organization may have its own customized format according to their own needs. Once the test plan is ready and application is under development; testers design and document the test cases. In the “Application Testing – Methodologies” section above, I have listed the TC design techniques.

Application Testing Cycles

Once the AUT is ready for testing, the practical phase of testing cycle starts in which testers actually execute the test cases on AUT. Keep in mind that here the testing cycle is discussed regardless of Testing Levels (Unit, Module, Integration, System and User Acceptance) and Testing Environments (Dev, QA, Client’s Replica, Live).

a. Smoke Testing: The very first testing cycle that is wide and shallow in approach. The purpose of smoke testing is to verify that there are no crashes in the application and it is suitable for further testing.

b. Sanity Testing: The second testing cycle that is narrow and deep in its approach. Its purpose is to verify that a specific module is working properly and is suitable for complete testing.

Tip: Usually there is not ample amount of time available to run these two cycles separately. So, a mixture of both these cycles is adopted in practical.

c. Functional Testing: The proper and full fledged testing of application is performed in this cycle. The primary focus of this activity is to verify the business logic of the application.

d. Regression Testing: This is the final cycle of testing in which the bug-fixes and/or updates are verified. Moreover, regression testing also ensures that there is no malfunctioning in other areas of AUT due to fixes and changes.

Bugs are logged in every testing cycle. There is no distinct border line between the testing cycles. For example, in Regression the Functionality is also verified and it may also require smoke, sanity or their merger first.

Application Testing – Best Practices

I think, hundreds of articles are available about this on internet. Every article suggests different number of best practices ranging from 7 to 30 (that I have seen so far). However, I have just 5 tips for readers.

1. Plan Properly
2. Test Keenly
3. Log the bugs Clearly
4. Do Regression Test Efficiently
5. Improve above four skills Continuously

Conclusion: Application Testing is a vast subject and the primary activity of any software tester. In this article, I have provided the overview of some most fundamental and necessary areas that fall under this topic. Application Testing involves strategies, phenomena, approaches, tools, technologies and guidelines. However, I have addressed the conceptual and practical insight of its salient concerns.

---------------------
I just wanted to thank you for subscribing to SoftwareTestingHelp.com. This blog is a success because of you and your support. Here's to your success!

As always, I appreciate your valuable comments. Visit below link to post your comment:
Application Testing – Into the Basics of Software Testing!

Thank you so much! ---------------------
Recommended: TestLodge - Online test case management tool - Web based test case management software allowing you to manage your test plans, requirements, test cases and test runs with ease.

1) Performance Testing:

Performance testing is the testing, which is performed, to ascertain how the components of a system are performing, given a particular situation. Resource usage, scalability and reliability of the product are also validated under this testing. This testing is the subset of performance engineering, which is focused on addressing performance issues in the design and architecture of software product.

Performance Testing Goal:

The primary goal of performance testing includes establishing the benchmark behaviour of the system. There are a number of industry-defined benchmarks, which should be met during performance testing.

Performance testing does not aim to find defects in the application, it address a little more critical task of testing the benchmark and standard set for the application. Accuracy and close monitoring of the performance and results of the test is the primary characteristic of performance testing.

Example:

For instance, you can test the application network performance on Connection Speed vs. Latency chart. Latency is the time difference between the data to reach from source to destination. Thus, a 70kb page would take not more than 15 seconds to load for a worst connection of 28.8kbps modem (latency=1000 milliseconds), while the page of same size would appear within 5 seconds, for the average connection of 256kbps DSL (latency=100 milliseconds). 1.5mbps T1 connection (latency=50 milliseconds) would have the performance benchmark set within 1 second to achieve this target.

For example, the time difference between the generation of request and acknowledgement of response should be in the range of x ms (milliseconds) and y ms, where x and y are standard digits. A successful performance testing should project most of the performance issues, which could be related to database, network, software, hardware etc…

2) Load Testing:

Load testing is meant to test the system by constantly and steadily increasing the load on the system till the time it reaches the threshold limit. It is the simplest form of testing which employs the use of automation tools such as LoadRunner or any other good tools, which are available. Load testing is also famous by the names like volume testing and endurance testing.

The sole purpose of load testing is to assign the system the largest job it could possible handle to test the endurance and monitoring the results. An interesting fact is that sometimes the system is fed with empty task to determine the behaviour of system in zero-load situation.

Load Testing Goal:

The goals of load testing are to expose the defects in application related to buffer overflow, memory leaks and mismanagement of memory. Another target of load testing is to determine the upper limit of all the components of application like database, hardware and network etc… so that it could manage the anticipated load in future. The issues that would eventually come out as the result of load testing may include load balancing problems, bandwidth issues, capacity of the existing system etc…

Example:

For example, to check the email functionality of an application, it could be flooded with 1000 users at a time. Now, 1000 users can fire the email transactions (read, send, delete, forward, reply) in many different ways. If we take one transaction per user per hour, then it would be 1000 transactions per hour. By simulating 10 transactions/user, we could load test the email server by occupying it with 10000 transactions/hour.

3) Stress testing

Under stress testing, various activities to overload the existing resources with excess jobs are carried out in an attempt to break the system down. Negative testing, which includes removal of the components from the system is also done as a part of stress testing. Also known as fatigue testing, this testing should capture the stability of the application by testing it beyond its bandwidth capacity.

The purpose behind stress testing is to ascertain the failure of system and to monitor how the system recovers back gracefully. The challenge here is to set up a controlled environment before launching the test so that you could precisely capture the behaviour of system repeatedly, under the most unpredictable scenarios.

Stress Testing Goal:

The goal of the stress testing is to analyse post-crash reports to define the behaviour of application after failure. The biggest issue is to ensure that the system does not compromise with the security of sensitive data after the failure. In a successful stress testing, the system will come back to normality along with all its components, after even the most terrible break down.

Example:

As an example, a word processor like Writer1.1.0 by OpenOffice.org is utilized in development of letters, presentations, spread sheets etc… Purpose of our stress testing is to load it with the excess of characters.

To do this, we will repeatedly paste a line of data, till it reaches its threshold limit of handling large volume of text. As soon as the character size reaches 65,535 characters, it would simply refuse to accept more data. The result of stress testing on Writer 1.1.0 produces the result that, it does not crash under the stress and that it handle the situation gracefully, which make sure that application is working correctly even under rigorous stress conditions.

Further reading – Web Application Load, Stress and Performance Testing Using WAPT.

Have queries on Software Testing? All you need to do is post your questions in the comment section below.

---------------------
I just wanted to thank you for subscribing to SoftwareTestingHelp.com. This blog is a success because of you and your support. Here's to your success!

As always, I appreciate your valuable comments. Visit below link to post your comment:
Difference Between Performance Testing, Load Testing and Stress Testing – With Examples

Thank you so much! ---------------------
Recommended: TestLodge - Online test case management tool - Web based test case management software allowing you to manage your test plans, requirements, test cases and test runs with ease.

Installation testing is like introducing a guest in your home. The new guest should be properly introduced to all the family members in order to feel him comfortable. Installation of new software is also quite like above example.

If your installation is successful on the new system then customer will be definitely happy but what if things are completely opposite. If installation fails then our program will not work on that system not only this but can leave user’s system badly damaged. User might require to reinstall the full operating system.

In above case will you make any impression on user? Definitely not! Your first impression to make a loyal customer is ruined due to incomplete installation testing. What you need to do for a good first impression? Test the installer appropriately with combination of both manual and automated processes on different machines with different configuration. Major concerned of installation testing is Time! It requires lot of time to even execute a single test case. If you are going to test a big application installer then think about time required to perform such a many test cases on different configurations.

We will see different methods to perform manual installer testing and some basic guideline for automating the installation process.

To start installation testing first decide on how many different system configurations you want to test the installation. Prepare one basic hard disk drive. Format this HDD with most common or default file system, install most common operating system (Windows) on this HDD. Install some basic required components on this HDD. Each time create images of this base HDD and you can create other configurations on this base drive. Make one set of each configuration like Operating system and file format to be used for further testing.

How we can use automation in this process? Well make some systems dedicated for creating basic images (use software’s like Norton Ghost for creating exact images of operating system quickly) of base configuration. This will save your tremendous time in each test case. For example if time to install one OS with basic configuration is say 1 hour then for each test case on fresh OS you will require 1+ hour. But creating image of OS will hardly require 5 to 10 minutes and you will save approximately 40 to 50 minutes!

You can use one operating system with multiple attempts of installation of installer. Each time uninstalling the application and preparing the base state for next test case. Be careful here that your uninstallation program should be tested before and should be working fine.

Installation testing tips with some broad test cases:

1) Use flow diagrams to perform installation testing. Flow diagrams simplify our task. See example flow diagram for basic installation testing test case.

Add some more test cases on this basic flow chart Such as if our application is not the first release then try to add different logical installation paths.

2) If you have previously installed compact basic version of application then in next test case install the full application version on the same path as used for compact version.

3) If you are using flow diagram to test different files to be written on disk while installation then use the same flow diagram in reverse order to test uninstallation of all the installed files on disk.

4) Use flow diagrams to automate the testing efforts. It will be very easy to convert diagrams into automated scripts.

5) Test the installer scripts used for checking the required disk space. If installer is prompting required disk space 1MB, then make sure exactly 1MB is used or whether more disk space utilized during installation. If yes flag this as error.

6) Test disk space requirement on different file system format. Like FAT16 will require more space than efficient NTFS or FAT32 file systems.

7) If possible set a dedicated system for only creating disk images. As said above this will save your testing time.

8 ) Use distributed testing environment in order to carry out installation testing. Distributed environment simply save your time and you can effectively manage all the different test cases from a single machine. The good approach for this is to create a master machine, which will drive different slave machines on network. You can start installation simultaneously on different machine from the master system.

9) Try to automate the routine to test the number of files to be written on disk. You can maintain this file list to be written on disk in and excel sheet and can give this list as a input to automated script that will check each and every path to verify the correct installation.

10) Use software’s available freely in market to verify registry changes on successful installation. Verify the registry changes with your expected change list after installation.

11) Forcefully break the installation process in between. See the behavior of system and whether system recovers to its original state without any issues. You can test this “break of installation” on every installation step.

12) Disk space checking: This is the crucial checking in the installation-testing scenario. You can choose different manual and automated methods to do this checking. In manual methods you can check free disk space available on drive before installation and disk space reported by installer script to check whether installer is calculating and reporting disk space accurately. Check the disk space after the installation to verify accurate usage of installation disk space. Run various combination of disk space availability by using some tools to automatically making disk space full while installation. Check system behavior on low disk space conditions while installation.

13) As you check installation you can test for uninstallation also. Before each new iteration of installation make sure that all the files written to disk are removed after uninstallation. Some times uninstallation routine removes files from only last upgraded installation keeping the old version files untouched. Also check for rebooting option after uninstallation manually and forcefully not to reboot.

I have addressed many areas of manual as well as automated installation testing procedure. Still there are many areas you need to focus on depending on the complexity of your software under installation. These not addressed important tasks includes installation over the network, online installation, patch installation, Database checking on Installation, Shared DLL installation and uninstallation etc.

Hope this article will be a basic guideline to those having trouble to start with software installation testing both manually or in automation.

If you like this article you would also like to subscribe to our email newsletter.

---------------------
I just wanted to thank you for subscribing to SoftwareTestingHelp.com. This blog is a success because of you and your support. Here's to your success!

As always, I appreciate your valuable comments. Visit below link to post your comment:
Software Installation/Uninstallation Testing

Thank you so much! ---------------------
Recommended: TestLodge - Online test case management tool - Web based test case management software allowing you to manage your test plans, requirements, test cases and test runs with ease.

This is the method of verification. Verifying that the bugs are fixed and the newly added feature have not created in problem in previous working version of software.

Why regression Testing?
Regression testing is initiated when programmer fix any bug or add new code for new functionality to the system. It is a quality measure to check that new code complies with old code and unmodified code is not getting affected.
Most of the time testing team has task to check the last minute changes in the system. In such situation testing only affected application area in necessary to complete the testing process in time with covering all major system aspects.

How much regression testing?
This depends on the scope of new added feature. If the scope of the fix or feature is large then the application area getting affected is quite large and testing should be thoroughly including all the application test cases. But this can be effectively decided when tester gets input from developer about the scope, nature and amount of change.

What we do in regression testing?

• Rerunning the previously conducted tests
• Comparing current results with previously executed test results.

Regression Testing Tools:
Automated Regression testing is the testing area where we can automate most of the testing efforts. We run all the previously executed test cases this means we have test case set available and running these test cases manually is time consuming. We know the expected results so automating these test cases is time saving and efficient regression testing method. Extent of automation depends on the number of test cases that are going to remain applicable over the time. If test cases are varying time to time as application scope goes on increasing then automation of regression procedure will be the waste of time.

Most of the regression testing tools are record and playback type. Means you will record the test cases by navigating through the AUT and verify whether expected results are coming or not.
Example regression testing tools are:

• Winrunner
• QTP
• AdventNet QEngine
• Regression Tester
• vTest
• Watir
• Selenium
• actiWate
• Rational Functional Tester
• SilkTest

Most of the tools are both Functional as well as regression testing tools.

Regression Testing Of GUI application:
It is difficult to perform GUI(Graphical User Interface) regression testing when GUI structure is modified. The test cases written on old GUI either becomes obsolete or need to reuse. Reusing the regression testing test cases means GUI test cases are modified according to new GUI. But this task becomes cumbersome if you have large set of GUI test cases.

---------------------
I just wanted to thank you for subscribing to SoftwareTestingHelp.com. This blog is a success because of you and your support. Here's to your success!

As always, I appreciate your valuable comments. Visit below link to post your comment:
Regression Testing with Regression Testing Tools and methods

Thank you so much! ---------------------
Recommended: TestLodge - Online test case management tool - Web based test case management software allowing you to manage your test plans, requirements, test cases and test runs with ease.

Black box testing treats the system as a “black-box”, so it doesn’t explicitly use Knowledge of the internal structure or code. Or in other words the Test engineer need not know the internal working of the “Black box” or application.

Main focus in black box testing is on functionality of the system as a whole. The term ‘behavioral testing’ is also used for black box testing and white box testing is also sometimes called ‘structural testing’. Behavioral test design is slightly different from black-box test design because the use of internal knowledge isn’t strictly forbidden, but it’s still discouraged.

Each testing method has its own advantages and disadvantages. There are some bugs that cannot be found using only black box or only white box. Majority of the applicationa are tested by black box testing method. We need to cover majority of test cases so that most of the bugs will get discovered by blackbox testing.

Black box testing occurs throughout the software development and Testing life cycle i.e in Unit, Integration, System, Acceptance and regression testing stages.

Tools used for Black Box testing:
Black box testing tools are mainly record and playback tools. These tools are used for regression testing that to check whether new build has created any bug in previous working application functionality. These record and playback tools records test cases in the form of some scripts like TSL, VB script, Java script, Perl.

Advantages of Black Box Testing
- Tester can be non-technical.
- Used to verify contradictions in actual system and the specifications.
- Test cases can be designed as soon as the functional specifications are complete

Disadvantages of Black Box Testing
- The test inputs needs to be from large sample space.
- It is difficult to identify all possible inputs in limited testing time. So writing test cases is slow and difficult
- Chances of having unidentified paths during this testing

Methods of Black box Testing:

Graph Based Testing Methods:
Each and every application is build up of some objects. All such objects are identified and graph is prepared. From this object graph each object relationship is identified and test cases written accordingly to discover the errors.

Error Guessing:
This is purely based on previous experience and judgment of tester. Error Guessing is the art of guessing where errors can be hidden. For this technique there are no specific tools, writing the test cases that cover all the application paths.

Boundary Value Analysis:
Many systems have tendency to fail on boundary. So testing boundry values of application is important. Boundary Value Analysis (BVA) is a test Functional Testing technique where the extreme boundary values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values.

Extends equivalence partitioning
Test both sides of each boundary
Look at output boundaries for test cases too
Test min, min-1, max, max+1, typical values

BVA techniques:
1. Number of variables
For n variables: BVA yields 4n + 1 test cases.
2. Kinds of ranges
Generalizing ranges depends on the nature or type of variables
Advantages of Boundary Value Analysis
1. Robustness Testing – Boundary Value Analysis plus values that go beyond the limits
2. Min – 1, Min, Min +1, Nom, Max -1, Max, Max +1
3. Forces attention to exception handling

Limitations of Boundary Value Analysis
Boundary value testing is efficient only for variables of fixed values i.e boundary.

Equivalence Partitioning:
Equivalence partitioning is a black box testing method that divides the input domain of a program into classes of data from which test cases can be derived.

How is this partitioning performed while testing:
1. If an input condition specifies a range, one valid and one two invalid classes are defined.
2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined.
4. If an input condition is Boolean, one valid and one invalid class is defined.

Comparison Testing:
Different independent versions of same software are used to compare to each other for testing in this method.

Reference- http://www.softrel.org/stgb.html

---------------------
I just wanted to thank you for subscribing to SoftwareTestingHelp.com. This blog is a success because of you and your support. Here's to your success!

As always, I appreciate your valuable comments. Visit below link to post your comment:
Black Box Testing: Types and techniques of BBT

Thank you so much! ---------------------
Recommended: TestLodge - Online test case management tool - Web based test case management software allowing you to manage your test plans, requirements, test cases and test runs with ease.