What is Smoke Testing? Difference in Smoke Testing and Sanity Testing

Overview of Smoke Testing:

When the development team releases a build to the QA for testing, it is obviously not possible to test the entire build and verify immediately if any of the implementations is having bugs or if any of the working functionality is broken.

In the light of this, how will a QA make sure that the basic functionalities are working fine?

The answer to this will be to perform a Smoke Testing.

What You Will Learn:

Introduction

Smoke testing is not an exhaustive testing but it is a group of tests that are executed to verify if the basic functionalities of that particular build are working fine as expected or not. This is and should always be the first test to be done on any ‘new’ build.

Once the tests are marked as Smoke tests (in the test suite) pass, only then the build is accepted by the QA for in-depth testing and/or regression. If any of the smoke tests fail, the build is rejected and the development team needs to fix the issue and release a new build for testing.

Theoretically, Smoke test is defined as a surface-level testing to certify that the build provided by the development team to the QA team is ready for further testing. This testing is also performed by the development team before releasing the build to the QA team.

This testing is normally used in Integration testing, System testing, and Acceptance level testing. Never treat this as a substitute for the actual end to end complete testing. It comprises of both positive and negative tests depending on the build implementation.

Need of Smoke Testing with Examples

As mentioned above, smoke testing is normally used for Integration, System and Acceptance testing.

In my 8 years career as a QA, I always accepted a build only after I had performed a smoke test. So, let us understand what is smoke test from the perspective of all these three testing, with some examples.

#1) Acceptance Testing:

Whenever a build is released to the QA, smoke test in the form of an acceptance testing should be done.

In this test, the first and most important smoke test is to verify the basic expected functionality of the implementation. Like this, you should verify all the implementations for that particular build.

Let us take the following examples as in implementations done in a build to understand the smoke tests for those:

In the above build, at acceptance level, the smoke test will mean to verify that the basic three implementations are working fine. If any of these three is broken, then the QA should reject the build.

#2) Integration Testing:

This testing is usually done when the individual modules are implemented and tested. In the Integration level, this testing is performed to make sure that all the basic integration and end to end functionalities are working fine as expected.

It may be an integration of two modules or all modules together, hence the complexity of the smoke test will vary depending on the level of integration.

Let us take the following example of integration implementation for this testing:

In this build, the smoke test will not only verify these three basic implementations but for the third implementation, a few cases for the complete integration too. It helps a lot to find out the issues that get introduced in integration and the ones that went unnoticed by the development team.

#3) System Testing:

As the name itself suggests, for system level, the smoke testing includes tests for the most important and commonly used workflows of the system.  This is done only after the complete system is ready & tested, and this testing for system level can be referred as smoke testing before regression testing also.

Before starting the regression of the complete system, the basic end to end features is tested as a part of smoke test. The smoke test suite for the complete system comprises of the end to end test cases that the end users are going to use very frequently.

This is usually done with the help of automation tools.

Importance of Smoke Testing in SCRUM Methodology

Nowadays, the projects hardly follow the Waterfall methodology in project implementation, mostly all the projects follow Agile and SCRUM only. Compared to the traditional waterfall method, smoke testing holds high regards in SCRUM and Agile.

I worked for 4 years in SCRUM. And as we know that in SCRUM, the sprints are of shorter duration and hence it is of extreme importance to do this testing so that the failed builds can immediately be reported to the development team and fixed.

Following are some takeaway on the importance of this testing in SCRUM:

Relation of Smoke Testing to BAT

BAT stands for Build Acceptance Testing and smoke testing is directly related to build acceptance testing.

In BAT, we do the same testing – to verify if the build has not failed and that the system is working fine or not. Sometimes, it happens that when the build is created, some issues get introduced and when it is delivered, the build doesn’t work for the QA.

I would say that BAT is a part of smoke testing because if the system is failing how can you as a QA accept the build for testing? Not just the functionalities, the system itself has to work before the QA’s proceed with the in-depth testing.

Difference between Smoke and Sanity Testing

Most of the times we get confused between the meaning of Sanity testing and Smoke testing. First of all, these two testing are way “different” and performed during different stages of a testing cycle.



S. No.Smoke Testing



Sanity Testing



1Smoke testing means to verify (basic) that the implementations done in a build are working fine.Sanity testing means to verify the newly added functionalities, bugs etc. are working fine.
2This is the first testing on the initial build.Done when the build is relatively stable.
3Done on every build.Done on stable builds post regression.

Following is a diagrammatic representation of the difference:

SMOKE TESTING:

SANITY TESTING:

Smoke Test Cycle

The following flowchart explains the smoke testing cycle. Once a build is deployed to a QA, the basic cycle followed is that if the smoke test passes, the build is accepted by the QA team for further testing but if it fails, the build is rejected until the reported issues are fixed.

Smoke Test Cycle

Who Should  Perform Smoke Test?

Not the whole team is involved in this type of testing to avoid the wastage of time of all the QA’s.

Smoke testing is ideally performed by the QA lead who decides based on the the result as for whether to pass the build to the team for further testing or reject it. Or in the absence of the lead, the QA’s themselves can also perform this testing.

At times, when the project is a large scale one, a group of QA can also perform this testing to check for any showstoppers. But this is not so in the case of SCRUM because SCRUM is a flat structure with no Leads or Managers and each tester has their own responsibilities towards their stories.

Hence individual QA’s perform this testing for the stories that they own.

Why Should you Automate Smoke Tests?

This testing is the first test to be done on a build released by the development team(s). Based on the results of this testing, the further testing is done (or the build is rejected).

The best way to do this testing is to use a automation tool and schedule the smoke suite to run when a new build is created. You may be thinking why should I “automate the smoke testing suite”?

Let us look at the following case:

Let’s say that you are a week away from your release and out of the total 500 test cases, your smoke test suite comprises of 80-90. If you start executing all these 80-90 test cases manually, imagine how much time will you take? I think 4-5 days (minimum).

But if you use automation and create scripts to run all these 80-90 test cases then ideally, these will be run in 2-3 hours and you will have the results with you instantly. Didn’t it save your precious time and give you the results about the build in much less time?

5 years back, I was testing a financial projection app, which took inputs about your salary, savings etc., and projected your taxes, savings, profits depending on the financial rules. Along with this, we had customization for countries that depend on the country and its tax rules used to change (in the code).

For this project, I had 800 test cases and 250 were smoke test cases. With the use of Selenium, we could easily automate and get the results of those 250 test cases in 3-4 hours. It not only saved our time but showed us ASAP about the showstoppers.

Hence, unless it is impossible to automate, do take the help of automation for this testing.

Advantages and Disadvantages

Let us first take a look at the advantages as it has a lot to offer when compared to the few disadvantages.

Following are the Advantages:

Disadvantages:

Conclusion

Smoke testing should definitely be done on every build as it points out the major failures and showstoppers at a very early stage. This applies not only to new functionalities but also to the integration of modules, fixing of issues and improvisation as well. It is a very simple process to perform and get the correct result.

Smoke testing can be treated as the entry point for complete functional testing of a functionality or system (as a whole). But before that, the QA team should be very clear about what tests are to be done as smoke tests. This testing can minimize the efforts, save time and improve the quality of the system. It holds a very important place in sprints as the time in sprints is less.

Smoke testing can be done both manually and also with the help of automation tools. But the best and preferred way is to use automation tools to save time.