Continuous Testing in DevOps

What is Continuous Testing and Continuous Testing Pipeline in DevOps?

Hope you all enjoyed the last tutorial on Continuous Deployment in DevOps.

We know the importance of testing in any software delivery and DevOps being short cycle of deliveries, it is impossible to run all the designed test cases each time manually, when a single line of code is updated in the version control tool and that is where continuous testing and automated continuous testing pipeline come into the picture in DevOps.

Continuous Testing In DevOps

 Suggested Reading => DevOps Training Tutorial from Scratch

Benefits of CT:

    • Quality and speed are the huge benefits of CT.
    • Faster and quicker feedback on the code.
    • Boosts the confidence of the team and encourages them to improve continuously.


VIDEO Part 3 Block 4: Continuous Testing – 14 minutes 39 seconds


In this block, we will learn about Continuous testing and Continuous testing pipeline in detail.

Continuous testing is another important process of the continuous delivery pipeline along with Continuous Integration, in a pipeline, it includes, various testing phases wherein the automated tests are run along with the automated quality gates in between.

Thus, continuous testing is, executing automated tests, continuously and repeatedly against the code base and the various deployment environments.

Mainly, unit tests, static code analysis, security code analysis, integration tests, load and performance tests are part an of continuous testing which is run in an automated continuous testing pipeline.

As continuous integration and continuous deployment are called CI, CD, continuous testing is more often called as CT.

If you see this diagram, which is a continuous delivery pipeline, this pipeline includes two pipelines, one is a build pipeline which is CI pipeline or continuous integration pipeline, which consists of automated build trigger, compiling, building and deploying.

The other one is a Testing pipeline, which is a continuous testing pipeline

Now let’s see more on continuous testing.

We know the importance of testing, testing every line of code….. testing every time…and testing at different stages and it is almost impossible to run all of the designed tests each time manually when a line of code is updated into version control.

That’s where the continuous testing comes into the picture.

So, unless the code that gets into the automated continuous integrated pipeline, gets tested thoroughly and ensures the required quality, there is no use for releasing the software to the customers. I mean quality cannot be ensured unless the code is thoroughly tested.

So, continuous testing, as defined earlier is to execute various types of tests, continuously on the code base and on different environments that it gets deployed on to, as predefined and designed in the continuous delivery pipeline.

Like you see in the picture, unit tests happen on the CI server itself, which tests each unit of the system in isolation.

Integration tests happen on Integration environment which basically verifies the components integrated together. System tests in the system testing environment where the BIG system with all the integrated components and interfaces are tested through system-level scenarios in a system testing environment and so on.

And the depth of testing often progresses as the simulation of environment gets closer to production.

Continuous testing progressively becomes tougher and longer with the progression towards the production environment as we need to slowly add a number of tests and more complicated tests as the code matures and environment complexity advances.

It is not that the same test cases would be run throughout, the test cases need to be updated each time at different phases and automated scripts are updated, as the code becomes more matured, progresses to a higher level of environment where configurations and infrastructure also advance, till it gets into production.

So, even the time taken to run the tests increases as the testing progresses towards release point, like unit testing might take very less time to run whereas some integration tests or some system tests or load tests might take few long hours to run or might take few days to run.

Here the continuous testing would be mainly running the automated test cases automatically with a trigger. But as we defined earlier, continuous delivery also involves certain manual tests and gates, wherein certain tests are carried out manually, before pushing into production.

These intermediate quality gates at every stage of testing and increases the confidence in the code.

So, the continuous testing pipeline as such includes unit testing along with preliminary automated security verifications. Then gets into an integration level of testing, where automated integration tests are run, then on to a system level where system-level scenarios are automated and run.

Here even certain performance test scenarios are also carried out.

Then goes to the ‘Acceptance testing’ which basically includes the automated site acceptance test cases and then finally on to the ‘User Acceptance Testing’ which could be a manual execution and includes end-user participation to carry out the tests and a this will be a kind of final sign off on the product or a feature, where manual gate is invoked and finally deployed on to the production site.

So, basically, as continuous testing progresses, the complexity of tests and the test environment increases and gets to the environment which is closer to the production like simulation.

I need not specifically have to mention that all these test stages include build verification tests, sanity tests, smoke tests and regression tests as well, again as I said, it depends on what we design in the continuous testing and delivery pipeline.

This is the typical continuous testing pipeline, well it can be designed by the team based on the type of the product and the different levels of testing and types of testing that the product demands.

Continous testing requires integrating automation framework with the version control and CI tool and the various automated tools to carry out the functional and non-functional testing across different phases of testing, like:

  • Sonar for static code analysis,
  • Fortify for secure code analysis,
  • Selenium for functional testing,
  • Load runner for load testing etc.,

Microsoft TFS, Jenkins, chef, puppet are few tools that are available in the market to design the CI-CD pipeline.

But the thing is, these tools may not support the complete end to end automation, depending on the version control tool used, so few organizations may prefer to develop their own automation frameworks, which enables the end to end automation of the delivery pipeline from code commit to code delivery.

So, Continuous testing being a very crucial part of testing ensures the quality of the product or release and one should be very careful about selection of a tool, framework, etc., which primarily determines the quality and speed of delivery.

So set up of right continuous testing pipeline takes a bit more of time in the continuous delivery pipeline. Not just on the tool and framework part but also on the test cases part. Continuous testing also includes defining deployment pipeline within.

Because CT requires the automated deployment of the build on to various environments at different phases, which calls for automating the deployment and setting up the environments via automated scripts.

These automated scripts which include setting up infrastructure and environment configurations as a code are checked into version control tool and the Delivery pipeline picks it up from the version control tool to carry out the deployment. This is called the deployment pipeline.

Now lets come to the benefits of CT,

Achieving quality and speed is the biggest advantage of continuous testing.

Unlike earlier where testing used to happen only in the end, test throughout is the concept of continuous testing and hence the continuous testing in a delivery pipeline, allows the team to introduce quality gates anywhere and any number of quality gates, they want, in order to achieve the degree of quality that they need.

So, if at all, the code fails for testing at any particular point or gate in a pipeline, the team can go back and automatically fail the entire deployment up to that point.

This gives a clear indication to both the Dev and Ops team that something is missing there and the team can work upon to fix it. So, this is the advantage and flexibility of continuous testing pipeline.

So, the introduction of quality gates at various test phases governs the quality of the code better in the pipeline.

More the number of gates that the code passes, more will be the team’s confidence in the code that it can make it to the production at a higher level of quality.

So, continuous testing boosts the confidence of the team and encourages them to improve continuously.

Overall, if the team does not really neglect any of the test failures at any test phases or quality gates in the pipeline, definitely continuous testing will be a bonus towards achieving high-quality targets.

So to conclude on continuous testing, right from the unit tests that are run during the preliminary stage through the acceptance testing, performance testing and even certain manual tests that are going to be run are VERY VERY critical for defining continuous testing in the DevOps pipeline.

This completes our discussion on Part3 topics of continuous integration, continuous delivery, and continuous testing.

In our upcoming tutorial, we will discuss more on Configuration management, release management, and application performance monitoring.

PREV Tutorial | NEXT Tutorial