DevOps Testing Tutorial:
A recent RightScale survey has found that 54% of the companies have adopted DevOps and the interest around DevOps is increasing rapidly.
In this article, we will learn how this new software development methodology will impact QA and how the QA function as a whole should evolve to embrace this change.
In this article, we will learn more about DevOps and how it will impact QA and its functions.
What You Will Learn:
What is DevOps?
DevOps – is a combination of Development & Operations – it is a software development methodology which looks to integrate all the software development functions from development to operations within the same cycle.
This calls for higher level of coordination within the various stakeholders in the software development process (namely Development, QA & Operations)
An ideal DevOps cycle would start from:
- The dev writing code
- Building & deploying of binaries in a QA environment
- Executing test cases and finally
- Deploying onto Production in one smooth integrated flow.
Obviously, this approach places a great emphasis on automation of build, deployment, and testing. Use of Continuous Integration (CI) tools, automation testing tools become a norm in a DevOps cycle.
Although there are subtle differences between Agile and DevOps Testing, those working with Agile will find DevOps a little more familiar to work with (and eventually adopt). While Agile principles are applied successfully on the development & QA iterations, it is a different story altogether (and often a bone of contention) on the operations side. DevOps proposes to rectify this gap.
Now, instead of Continuous Integration, DevOps involves “Continuous Development”, where the code was written and committed to Version Control, will be built, deployed, tested and installed on the Production environment that is ready to be consumed by the end-user.
This process helps everyone in the entire chain since environments and processes are standardized. Every action in the chain is automated. It also gives freedom to all the stakeholders to concentrate their efforts on designing and coding a high-quality deliverable rather than worrying about the various building, operations, and QA processes.
It brings down the time-to-live drastically to about 3-4 hours, from the time code is written and committed, to deployment on production for end-user consumption.
In a nutshell, DevOps is an extension of Agile or I like to call it as“Agile on Steroids”.
Changed Role of QA in DevOps
Traditionally, QA would get a build which is deployed on their designated environment and QA would then commence their functional & regression testing. The build would ideally sit with the QA for a couple of days before the QA sign-off on the build. All these steps change in DevOps.
QA changes for DevOps Testing:
- QA are required to align their efforts in the DevOps cycle.
- They have to make sure that all their test cases are automated and achieve near 100% code coverage.
- They need to make sure that their environments are standardized and the deployment on their QA boxes are automated.
- All their pre-testing tasks, cleanups, post-testing tasks, etc. are automated and aligned with the Continuous Integration cycle.
As already mentioned, DevOps requires a high level of coordination between various functions of the deliverable chain. This also means that the boundaries between various roles of contributors in the chain become porous.
DevOps encourages everyone to contribute to the chain. So, amongst other things, a dev can configure deployments. Deployment engineers can add test cases to the QA repository. QA Engineers can configure their automation test cases into the DevOps chain.
Collectively, everyone in the chain is responsible for the quality and timeliness of the deliverables.
DevOps and Test Automation
To achieve such speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed in the QA environment. Specialized automation testing tools and continuous integration tools are used to achieve this integration. This also necessitates the building of a mature automation testing framework through which one can quickly script new test cases.
DevOps Testing Strategy: Tips for DevOps Success
- The test cases that are required to be executed for a particular build need to be identified.
- The test execution should essentially be lean.
- The QA and dev need to sit together and identify the areas affected due to a particular build and execute those related test cases plus a sanity test pass.
- You also need to configure specialized code analysis and coverage tools to make sure that you achieve near 100% code coverage.
- The concept of executing all regression test cases for a test pass is soon becoming obsolete.
- Strategy around testing new features need to be formalized and the interim builds can be supplied to QA who would, in turn, create test scripts and run these automation tests on the interim builds till the code becomes stable enough to be deployed onto the Production environment.
- All the environments required for testing need to be standardized and the deployments have to be automated.
- Using various automation techniques, QA should be able to fire automation testing runs across various cross-platform (and cross-browser in case of web applications) environments.
- Parallel execution of tests helps in reducing time-to-live, which in turn is the crux of a successful DevOps implementation.
- Exit criteria need to be set for each run so that when the results of the tests are fed back to the chain, a go/no-go decision to Production is taken.
- Blocker or Critical bugs found need to be reported & fixed and passed through the same chain of events before the code is deployed in the Production environment.
QA should also be able to detect problems early and report them proactively. To achieve this, they need to set up monitoring on the Production environment to be able to expose bugs before they cause a failure.
Setting up specialized counters like response times, memory & CPU utilization, etc. can provide a lot of insight into the end-user experience.
For example, if the average response time for login is gradually increasing over the various builds, QA should proactively report this issue for optimizing the login code, else future builds might cause end-user frustration due to high response times.
QA can also use a small subset of existing high priority test cases to be executed periodically on production, to actively monitor the environment. Bugs like, “This bug appears sometimes” or “Cannot Reproduce” can be caught through this strategy which, at the end, makes the application more stable and also gets more satisfied end-users.
Again, these monitors need to be configured to run automatically with rich reporting (like logs & screenshots of failures, etc.).
Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software development.
DevOps is the future. It’s a continuous improvement cycle that software development models undergo from time-to-time. You need to embrace, understand and inculcate it.
You need to master the various automation and continuous integration tools so that your automation efforts add value to the chain and are lean enough to quickly adapt to changes. You may be working on projects that may involve an alpha, beta and UAT environments before being deployed in the production environment.
The concept essentially remains the same. Automation and more automation is the core of a successful DevOps cycle. But, as a QA you should also be able to draw a line as to how much automation is too much automation.
About the Author: Aniket Deshpande is working as a QA Manager at AFour Technologies, Pune and has been working in the software testing field for the past 9+ years in various domains and platforms. He is passionate about DevOps and is working as a consultant to guide organizations in adopting DevOps testing strategies. If you are interested in knowing more, or you are looking to implement DevOps and associated Test Approach in your organization, feel free to contact the author.
What do you think about DevOps Testing? Do you think by getting developers and operations folks to work together can benefit the project?
Let us know your comments/suggestions about this article.