DevOps Testing Tutorial: How DevOps will Impact QA Testing?

DevOps Testing Tutorial: A recent RightScale survey has found that 54% of the companies have adopted DevOps and the interest around DevOps is increasing rapidly.

In this article, we will learn how this new Software Development methodology will impact QA and how the QA function as a whole should evolve to embrace this change.

          Check out => Complete DevOps Tutorial Series

In this article, we will learn more about DevOps and how it will impact QA and its functions.

How DevOps will Impact QA

What Is DevOps?

DevOps – is a combination of Development & Operations – it is a Software Development methodology that looks to integrate all the Software Development functions from development to operations within the same cycle.

This calls for a higher level of coordination within the various stakeholders in the software development process (namely Development, QA & Operations)

DevOps Cycle


An ideal DevOps cycle would start from:

  • The Dev writing code
  • Building & deploying binaries in a QA environment
  • Executing test cases and finally
  • Deploying onto Production in one smooth integrated flow.

Obviously, this approach places a great emphasis on automation of Build, Deployment, and Testing. Use of Continuous Integration (CI) tools, Automation Testing tools become a norm in a DevOps cycle.

Why DevOps?

Although there are subtle differences between Agile and DevOps Testing, those working with Agile will find DevOps a little more familiar to work with (and eventually adopt). While Agile principles are applied successfully in the development & QA iterations, it is a different story altogether (and often a bone of contention) on the operations side. DevOps proposes to rectify this gap.

Now, instead of Continuous Integration, DevOps involves “Continuous Development”, where the code was written and committed to Version Control, will be built, deployed, tested and installed on the Production environment that is ready to be consumed by the end-user.

This process helps everyone in the entire chain since environments and processes are standardized. Every action in the chain is automated. It also gives freedom to all the stakeholders to concentrate their efforts on designing and coding a high-quality deliverable rather than worrying about the various building, operations, and QA processes.

It brings down the time-to-live drastically to about 3-4 hours, from the time code is written and committed, to deployment on production for end-user consumption.

In a nutshell, DevOps is an extension of Agile or I like to call it“Agile on Steroids”.

Changed Role Of QA In DevOps

Traditionally, QA would get a build which is deployed in their designated environment and QA would then commence their Functional & Regression testing. The build would ideally sit with the QA for a couple of days before the QA sign-off on the build. All these steps change in DevOps.

QA changes for DevOps Testing:

  • QA are required to align their efforts in the DevOps cycle.
  • They have to make sure that all their test cases are automated and achieve near 100% code coverage.
  • They need to make sure that their environments are standardized and the deployment on their QA boxes is automated.
  • All their pre-testing tasks, cleanups, post-testing tasks, etc. are automated and aligned with the Continuous Integration cycle.

As already mentioned, DevOps requires a high level of coordination between various functions of the deliverable chain. This also means that the boundaries between various roles of contributors in the chain become porous.

DevOps encourages everyone to contribute to the chain. So, amongst other things, a dev can configure deployments. Deployment engineers can add test cases to the QA repository. QA Engineers can configure their automation test cases into the DevOps chain.

Collectively, everyone in the chain is responsible for the quality and timeliness of the deliverables.

DevOps and Test Automation

To achieve such speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed in the QA environment. Specialized Automation Testing tools and continuous integration tools are used to achieve this integration.

This also necessitates the building of a mature Automation Testing framework through which one can quickly script new test cases.

DevOps Testing Strategy: Tips for DevOps Success

  • The test cases that are required to be executed for a particular build need to be identified.
  • The test execution should essentially be lean.
  • The QA and Dev need to sit together and identify the areas affected due to a particular build and execute those related test cases plus a sanity test pass.
  • You also need to configure specialized code analysis and coverage tools to make sure that you achieve near 100% code coverage.
  • The concept of executing all regression test cases for a test pass is soon becoming obsolete.
  • The strategy around testing new features needs to be formalized and the interim builds can be supplied to QA who would, in turn, create test scripts and run these automation tests on the interim builds till the code becomes stable enough to be deployed onto the Production environment.
  • All the environments required for testing need to be standardized and the deployments have to be automated.
  • Using various automation techniques, QA should be able to fire Automation Testing runs across various cross-platform (and cross-browser in case of web applications) environments.
  • Parallel execution of tests helps in reducing time-to-live, which in turn is the crux of a successful DevOps implementation.
  • Exit criteria need to be set for each run so that when the results of the tests are fed back to the chain, a go/no-go decision to Production is taken.
  • Blocker or Critical bugs found need to be reported & fixed and passed through the same chain of events before the code is deployed in the Production environment.

Application Monitoring

QA should also be able to detect problems early and report them proactively. To achieve this, they need to set up monitoring on the Production environment to be able to expose bugs before they cause a failure.

Setting up specialized counters like response times, memory & CPU utilization, etc. can provide a lot of insight into the end-user experience.

For Example, if the average response time for login is gradually increasing over the various builds, QA should proactively report this issue for optimizing the login code, else future builds might cause end-user frustration due to high response times.

QA can also use a small subset of existing high priority test cases to be executed periodically on production, to actively monitor the environment. Bugs like, “This bug appears sometimes” or “Cannot Reproduce” can be caught through this strategy which, in the end, makes the application more stable and also gets more satisfied end-users.

Again, these monitors need to be configured to run automatically with rich reporting (like logs & screenshots of failures, etc.).


Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software development.

DevOps is the future. It’s a continuous improvement cycle that software development models undergo from time-to-time. You need to embrace, understand and inculcate it.

You need to master the various automation and continuous integration tools so that your automation efforts add value to the chain and are lean enough to quickly adapt to changes. You may be working on projects that may involve alpha, beta and UAT environments before being deployed in the production environment.

The concept essentially remains the same. Automation and more automation is the core of a successful DevOps cycle. But, as a QA you should also be able to draw a line as to how much automation is too much automation.

About the Author: Aniket Deshpande is working as a QA Manager at AFour Technologies, Pune and has been working in the software testing field for the past 9+ years in various domains and platforms. He is passionate about DevOps and is working as a consultant to guide organizations in adopting DevOps testing strategies.

If you are interested in knowing more, or you are looking to implement DevOps and associated Test Approach in your organization, feel free to contact the author.

What do you think about DevOps Testing? Do you think by getting developers and operations folks to work together can benefit the project?

Let us know your comments/suggestions about this article.

PREV Tutorial | NEXT Tutorial

Recommended Reading

45 thoughts on “DevOps Testing Tutorial: How DevOps will Impact QA Testing?”

  1. Article is really nice, but only question I have is what if product does not support Automation tool, then what can be done ?

    • Hi Siddharth,
      I feel there can be no product which cannot support automation, as all the web, desktop and mobile are covered. But if you still have something that cannot be automated, you can still implement devops as QA is a part of devops and the continuous integration part can still be implemented and your test team can receive builds for testing asap. As far as continuous delivery is concerned, it still needs some work as there might be other tasks like ORT etc that need to be done before sending the product to the customers for usage

  2. The Article is Very Nice and more Automation takes huge time to comlete the QA Cycle and revert back to Dev team for fixing bugs….

  3. Hi Siddharth Spehia,
    Automation is the corner-stone of a successful DevOp cycle. There are various ways in which automation can be achieved like Non-UI automation or API automation, etc. Obviously, you cannot have all your tests automated, and DevOps does not mandate this. But all your high-priority, end-to-end tests need to be automated and added to the cycle. Talk to your developers to help you achieve this. Ideally, you should start your automation scripting along with early dev builds so that you can provide feedback to the developers regarding blockers for automation.

  4. Hi Smriti,
    Apart from the regular automation testing tools you can take a look at some of the Continuous Integration tools, like Jenkins, TeamCity, Team Foundation Server, etc. You should be able to configure your automation runs through these tools.

  5. Smriti, since we’ve been working on Deployment Manager (a release automation tool) at Red Gate we’ve found that DevOps and QA (or testers) have more and more overlap. For instance we’ve discovered that a lot of our customers tend to set up environments specificially for QA and/or DevOps that they use to pass deployment packages to each other.

  6. Nice Article. Role of Developers, QA and Operations has changed because of Agile, and DevOps is one of the results.

    Automation is the key for a successful delivery, with enough instrumentation and alerts to have issues in the system reported as soon as possible.

  7. Hi..
    Nice article.. I would like to see the feedback loop illustrated in the process diagram. Does the cycle happen in one Sprint?
    I am trying to understand how different is this from Agile combined with process improvements made by teams over a period of time. Have we just standardized the same and termed it as DevOps?


  8. Hi Girish,
    Thanks for the feedback.
    You can call DevOps as more of a philosophy than a methodology and it imbibes most of its processes from Agile, which in-turn is more of a philosophy than a methodology. I guess you are right in a way when you say that it is “Agile combined with process improvements made by teams over a period of time”. But things like environment standardisation, full automation of the entire build-test-deploy cycle up to Production are somethings that DevOps tries to address. DevOps makes Agile more agile.

    P.S.: To answer your question, yes, this cycle happens in one Sprint.

  9. Good article Aniket…
    It would be so kind of you if you can tell us how DevOps can address the two challenges we face in agile or any paced up environment:
    1.Frequently changing requirements till system stabilisation.
    2.Automating initial round of testing in less time.

  10. Hi Hari,
    You need to start automating from the initial cycles. This will help you in achieving your automation milestones sooner.
    Also, if there are frequent changes in requirements, dev would require multiple builds to stabilise their code. QA can also use these interim builds to stabilise their automation code before it is finally Production ready.
    Again, close coordination and communication between the dev & QA team is the key…

  11. Nice article!!! :)
    However one doubt….BA will forward USER STORIES to developers , in the diagram it show BUGS –> DEV .can some please explain me .

  12. Hi Swapnil,
    Thanks for the feedback.
    Developers will either fix bugs or implement new features (User Stories). That is what is depicted in the diagram. Essentially, anyone in the chain can raise a bug (not just QAs).

  13. Hi,
    One doubt is how we can use this for 1st sprint where we will not be having for application during 1st deployment. So we cannot automate our test cases for 1st sprint till deployment is done. So I wanted to know how we can integrate automated test cases in “Devops” for 1st sprint.

    • Write your BDD feature file when you know the scenario after reading requirement
      At first run it will fail
      then refactor by implementing the step def and page object
      it will still fail if the developement is not done
      get the object details and update the object repo.
      when development is done; run your test should pass.

  14. Nice read. Came across the term DevOps during my SAFe Agilist workshop. Your article gave more insight into it. Thanks Aniket :)

  15. Hi,
    Thanks. Wonderful article to understand the DevOps which is future. Two Questions if you can help please.
    1. Early automation sounds good but in actual, while requirements are changing frequently, there will be too much rework in updating the automation scripts.
    Generally automation is advisable when system is matured. How beneficial it would be ?
    2. How can we automate test cases for first sprint when application is not ready ? To achieve this, I think FSDs should be detailed enough but again that is difficult when requirements are changing frequently.

    • Thanks Saurabh. I have same questions the moment I heard about DevOps. I would like to add third point
      3. What would be the solution if some Test Scenarios can not be automated?

      Sorry to switch over the topic. Very similarly when Agile is adopted, the I was searching for the answer for

      In which Sprint one can do the E2E business process testing?. For an example, like “Order to cash business process” in the typical enterprise setup where multiple applications are involved. Especially when the user stories for a release requires the code change only few application in the stack of application to complete E2E business process. In some places, they run different sprint for E2E (Runs once in every four Sprints or where there is a logical end) and others run after the last sprint like waterfall model. I feel this defeat the purpose of Agile. Any one has better solution?

  16. Article is good . With all these terminologies in market, what I feel are teams are coming closer and with co-ordinaton work is executed in a better way.

    Tomorrow in market 1 new terminology will come where BA will also sit with testers, then you will have excellent results

  17. Hi Aniket,
    We are a team of 5 developers working for a telecom client project. We are to give integrated devops as a part of the deliverable along with the work assigned. I need to know whether in a devops culture are the developers supposed to do QA as well, Write Test cases themselves & also write the selenium automation scripts themselves for CI CD? What does devops principles say? Should there be a dedicated Dev & QA team along with Devops engineers who could wear bot the hats of Dev & QA. Are the Devops engineers part of Dev Team or QA team or are shared resources or are they a dedicated team separate from Dev & QA teams?
    Now since I have just 5 developers should I train them in QA & in Devops automation & make them do all the work of Dev QA & Dev ops? Its going to be tightfisted situation if they are to deliver in weekly sprints as well as write QA test cases & do automation in the same week for the sprint. Should I propose the Client to allow me to add a team of QA engineers who know selenium & also know automation & let the 5 developers focus on Dev stories. Ofcuorse once the QA comes in then the Dev will start learning Selinium & contributing to the QA & the automation part.. Will this proposal be violating Devops principles that the developer needs to do QA & automation both in a typical Devops setup? Please suggest how to deal with this situation & what exactly are the devops guidelines for the duties of a developer, a QA & automation team. Or is it just one Devops team which does all 3 things?? Or is it a dedicated Devops team of engineers who take care od this integration of Dev QA & Deployment?? should the devops engineers be a part of Dev or QA team or are they a part of different devops team… I am really confused when it comes to resource allocation & resource billing for the client in a typice devops setup. Please advise.

  18. Very interesting document, anyway there is still a misusage of the word QA, and test and abuse of the word test automation. QA is validation, test is verification, proactivity vs. reactivity. If you want to achieve a DevOps establishment you should rethink the way you write the software, modularity independency and resilience are the key factor, then you think towards BDD, you will get your software automatically tested. If you have for example 10 agile teams each working in a separate product using LeSS framework, then I would like to understand how you ensure the the final product match with the consumer expectation and how you want to improve such a product. In this case to me QA means sit close to product owners and understand their expectation and proactively work to collect and maintain the focus on the roadmap, looking if the mvp match with the consumer expectation. Test automation and all the means to ensure the bugs are caught during the implementation it is part of the software development process, QA will give the methodology and the structure, DevOps team are implementing it and collaborating with QA to improve it.

  19. Wow this is amazing Devops development and operations i hope This concept will change the Future of web development and Testing.

  20. Hi Aniket,

    its nice blog those who want to switch their career as a Devops QA Role which is hot skills in the current market.

    Please let me know your email id so that i need to discuss more on Devops.

    Thanks a lot !

  21. Thanks for giving a great information about DevOps Good Explination nice Article
    anyone want to learn advance devops tools or devops online training

  22. Devops calls for a higher level of coordination within the various stakeholders in the software development process.

  23. ‘QA changes for DevOps Testing:’ – not sure how is this different to any other Dev Methodology. Regardless if this is Waterfall or Agile-like Methodlogies – you want to automate deployments/testing/clean-ups to the best extent you can. Needless to say that environments have to be standartized as well.
    So what is so specific here for DevOps? Just trying to the uathors point of view.

  24. Great work, thank you for providing this valuable information in the form of tutorials, it is awesome. I have been clicking ads here on purpose so that you can get paid for the Excellent job you have done. Thank you.

  25. hello,
    Since you are a QA Manager, do you have separate QA/Testers in the DevOps setup? Or this is shared on a Developer only? Meaning These Developers are multi-skilled at programming and testing. They can code review each other’s tasks and test each other’s tasks as well.
    Have you experienced a setup where in DevOps, there are still separate roles on BA, Dev & QA?

  26. Hi Aniket,

    Thanks for the article .

    On a project where a new system is being built and you have a large number of user stories / business requirements, along with functional specification and business rules, traditionally we (Software Testing function) would end up with a large number of Test cases / Test Scripts (say for example 650 tests scripts)
    This would require:
    -We write the Test cases,
    -Get them reviewed and signed-off
    -we develop all Test cases into Test scripts
    With a team of 4 we can achieve the above in circa 20 days

    -How do we achieve the above quicker than 20 days if we are to adopt automation?

    -Time taken to decide which tools to use for automation
    -Time it takes for Test Automation Specialists to understand/learn the system being built
    -Time it takes for Test Automation Spcialist write the 650 Tests and get them reviewed
    -Time taken to build and test the automation framework
    -Time taken to automate the 650 tests and to test them to ensure automated code works
    -Time taken to run the tests (this is the value added, but it seems an awful long road to get here)

  27. Hello again Aniket,

    Thanks for the article .

    How does DevOps work with System Integration Testing and also User Acceptance Testing?

    I recognise the logic and need to automate, but we seem to be selling it as a magic pill that instantly shortens manual testing effort and I just cant seem to see that.

    Software Testing on a new enterprise wide system being built requires extensive test preparation activities most of which will not immediately benefit from automation

    What are your thoughts?

  28. The Philosophy of Continuous integration (CI) is to make small code changes and check-in code frequently to a central repository and then ensuring that you are making progress in terms of features (or bug fixes) while not breaking any existing functionality. To be able to check if no existing functionality is broken is to check this frequently via automated tests. Thus, CI can meaningfully exist only when there is adequate automated testing.

  29. The Philosophy of Continuous integration (CI) is to make small code changes and check-in code frequently to a central repository and then ensuring that you are making progress in terms of features (or bug fixes) while not breaking any existing functionality. To be able to check if no existing functionality is broken is to check this frequently via automated tests. Thus, CI can meaningfully exist only when there is adequate automated testing.

  30. As DevOps brings in more and more automation to the fore, the need for traditional manual QA will decline sharply. However, manual functional QA would not cease to exist.


Leave a Comment