“DevOps” an Extension of Agile Methodology – How It will Impact QA?

DevOps and Software Testing:

A recent RightScale survey has found that 54% of companies have adopted DevOps and the interest around DevOps is increasing rapidly. In this article we will learn how this new software development methodology will impact QA and how the QA function as a whole should evolve to embrace this change.

What is DevOps?

DevOps – a combination of Development & Operations – is a software development methodology which looks to integrate all the software development functions from development to operations within the same cycle.

This calls for higher level of coordination within the various stakeholders in the software development process (namely Development, QA & Operations)

DevOps Cycle:

DevOps-Flow

So an ideal DevOps cycle would start from:

  • The dev writing code
  • Building & deploying of binaries on a QA environment
  • Executing test cases and finally
  • Deploying on to Production in one smooth integrated flow.

Obviously, this approach places great emphasis on automation of build, deployment and testing. Use of Continuous Integration (CI) tools, automation testing tools become a norm in a DevOps cycle.

Why DevOps?

Although there are subtle differences between Agile & DevOps, those working with Agile will find DevOps a little more familiar to work with (and eventually adopt). While Agile principles are applied successfully on the development & QA iterations, it is a different story altogether (and often a bone of contention) on the operations side. DevOps proposes to rectify this gap.

Now, instead of Continuous Integration, DevOps involves “Continuous Development”, whereby, code written and committed to Version Control, will be built, deployed, tested and installed on the Production environment ready to be consumed by the end-user.

This process helps everyone in the entire chain since environments and processes are standardized. Every action in the chain is automated. It also gives freedom to all stakeholders to concentrate their efforts on designing and coding a high quality deliverable rather than worrying about the various building, operations and QA processes.

It brings down the time-to-live drastically to about 3-4 hours, from the time code is written and committed, to deployment on production for end-user consumption.

In a nutshell, DevOps is an extension of Agile, or, as I like to call it, “Agile on Steroids”.

Changed Role of QA in DevOps

Traditionally, QA would get a build which is deployed on their designated environment and QA would then commence their functional & regression testing. The build would ideally sit with the QA for a couple of days before the QA sign-off on the build. All this changes in DevOps.

QA changes for DevOps:

  • QA are required to align their efforts in the DevOps cycle.
  • They have to make sure that all their test cases are automated and achieve near 100% code coverage.
  • They need to make sure that their environments are standardized and deployment on their QA boxes is automated.
  • All their pre-testing tasks, cleanups, post-testing tasks, etc. are automated and aligned with the Continuous Integration cycle.

As already mentioned, DevOps requires high level of coordination between various functions of the deliverable chain. What this also means is that boundaries between the various roles of contributors in the chain become porous.

DevOps encourages everyone to contribute across the chain. So, amongst other things, a dev can configure deployments. Deployment engineers can add test cases to the QA repository. QA Engineers can configure their automation test cases into the DevOps chain.

Collectively, everyone in the chain is responsible for the quality and timeliness of the deliverables.

DevOps and Test Automation

To achieve such speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed on the QA environment. Specialized automation testing tools and continuous integration tools are used to achieve this integration. This also necessitates building of a mature automation testing framework through which one can quickly script new test cases.

Tips for DevOps Success:

------------

  • The test cases that are required to be executed for a particular build need to be identified.
  • The test execution should essentially be lean.
  • The QA and dev need to sit together and identify the areas affected due to a particular build and execute those related test cases plus a sanity test pass.
  • You also need to configure specialized code analysis and coverage tools to make sure you achieve near 100% code coverage.
  • The concept of executing all regression test cases for a test pass is soon becoming obsolete.
  • Strategy around testing new features need to be formalized and interim builds can be supplied to QA who would in turn create test scripts and run these automation tests on these interim builds till the code becomes stable enough to be deployed on to Production environment.
  • All the environments required for testing need to be standardized and deployments have to be automated.
  • Using various automation techniques, QA should be able to fire automation testing runs across various cross-platform (and cross-browser in case of web applications) environments.
  • Parallel execution of tests helps in reducing time-to-live which is the crux of a successful DevOps implementation.
  • Exit criteria need to be set for each run so that when the results of the tests are fed back to the chain, a go/no-go decision to Production is taken.
  • Blocker or Critical bugs found need to be reported and fixed and passed through the same chain of events before the code is deployed on to the Production environment.

Application Monitoring

QA should also be able to detect problems early and report them proactively. To achieve this, they need to set up monitoring on the Production environment to be able to expose bugs before they cause a failure.

Setting up specialized counters like response times, memory & CPU utilization, etc. can provide a lot of insight into the end-user experience.

For example, if the average response time for login is gradually increasing over the various builds, QA should proactively report this issue for optimizing the login code, else future builds might cause end-user frustration due to high response times.

QA can also use a small subset of existing high priority test cases to be executed periodically on production to actively monitor the environment. Bugs like, “This bug appears sometimes” or “Cannot Reproduce” can be caught through this strategy which, in the end makes the application more stable and also gets more satisfied end-users.

Again, these monitors need to be configured to run automatically with rich reporting (like logs & screenshots of failures, etc.).

Conclusion

Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software development. DevOps is the future. It’s a continuous improvement cycle that software development models undergo from time-to-time. You need to embrace, understand and inculcate it.

You need to master the various automation and continuous integration tools, so that your automation efforts add value to the chain and are lean enough to quickly adapt to changes. You may be working on projects that may involve an alpha, beta and UAT environments before being deployed on the production environment.

The concept essentially remains the same. Automation, automation and more automation is the core of a successful DevOps cycle. But, as a QA you should also be able to draw a line as to how much automation is too much automation.

About the Author: Aniket Deshpande is working as a QA Manager at AFour Technologies, Pune and has been working in the software testing field for the past 9+ years in various domains and platforms. He is passionate about DevOps and is working as a consultant to guide organizations in adopting DevOps strategies. If you are interested in knowing more, or you are looking to implement DevOps and associated Test Approach in your organization, feel free to contact author.

What do you think about DevOps? Do you think by getting developers and operations folks to work together can benefit the project?



The Best Software Testing Training You'll Ever Get!

software testing QA training

17 comments ↓

#1 Siddharth Spehia on 11.08.13 at 3:00 am

Article is really nice, but only question I have is what if product does not support Automation tool, then what can be done ?

#2 Pravin on 11.08.13 at 3:56 am

This article is very useful.

#3 Saagar Bhatt - QA Magic on 11.08.13 at 6:22 am

Looks Great, A very Useful information…!!

#4 Vamsi on 11.08.13 at 7:55 am

The Article is Very Nice and more Automation takes huge time to comlete the QA Cycle and revert back to Dev team for fixing bugs….

#5 Smriti on 11.08.13 at 8:07 am

are there any tools which are used in DevOps and which are also need to be used by QAs?

#6 Aniket Deshpande on 11.08.13 at 1:13 pm

Hi Siddharth Spehia,
Automation is the corner-stone of a successful DevOp cycle. There are various ways in which automation can be achieved like Non-UI automation or API automation, etc. Obviously, you cannot have all your tests automated, and DevOps does not mandate this. But all your high-priority, end-to-end tests need to be automated and added to the cycle. Talk to your developers to help you achieve this. Ideally, you should start your automation scripting along with early dev builds so that you can provide feedback to the developers regarding blockers for automation.

#7 Aniket Deshpande on 11.08.13 at 1:15 pm

Hi Smriti,
Apart from the regular automation testing tools you can take a look at some of the Continuous Integration tools, like Jenkins, TeamCity, Team Foundation Server, etc. You should be able to configure your automation runs through these tools.

#8 Richard Muscat on 11.08.13 at 2:26 pm

Smriti, since we’ve been working on Deployment Manager (a release automation tool) at Red Gate we’ve found that DevOps and QA (or testers) have more and more overlap. For instance we’ve discovered that a lot of our customers tend to set up environments specificially for QA and/or DevOps that they use to pass deployment packages to each other.

#9 Icicletech on 11.08.13 at 2:50 pm

Nice Article. Role of Developers, QA and Operations has changed because of Agile, and DevOps is one of the results.

Automation is the key for a successful delivery, with enough instrumentation and alerts to have issues in the system reported as soon as possible.

#10 Amrutha on 11.08.13 at 7:54 pm

A great article.

Thank you so much for sharing on web.

#11 Girish on 11.10.13 at 2:43 am

Hi..
Nice article.. I would like to see the feedback loop illustrated in the process diagram. Does the cycle happen in one Sprint?
I am trying to understand how different is this from Agile combined with process improvements made by teams over a period of time. Have we just standardized the same and termed it as DevOps?

Cheers,
Girish

#12 Mansoor Shaikh on 11.11.13 at 5:51 am

Excellent article. Thanks Aniket for writing this article.

#13 Aniket Deshpande on 11.12.13 at 6:47 am

Hi Girish,
Thanks for the feedback.
You can call DevOps as more of a philosophy than a methodology and it imbibes most of its processes from Agile, which in-turn is more of a philosophy than a methodology. I guess you are right in a way when you say that it is “Agile combined with process improvements made by teams over a period of time”. But things like environment standardisation, full automation of the entire build-test-deploy cycle up to Production are somethings that DevOps tries to address. DevOps makes Agile more agile.

P.S.: To answer your question, yes, this cycle happens in one Sprint.

#14 Hari on 11.12.13 at 10:12 am

Good article Aniket…
It would be so kind of you if you can tell us how DevOps can address the two challenges we face in agile or any paced up environment:
1.Frequently changing requirements till system stabilisation.
2.Automating initial round of testing in less time.

#15 Aniket Deshpande on 11.12.13 at 2:48 pm

Hi Hari,
You need to start automating from the initial cycles. This will help you in achieving your automation milestones sooner.
Also, if there are frequent changes in requirements, dev would require multiple builds to stabilise their code. QA can also use these interim builds to stabilise their automation code before it is finally Production ready.
Again, close coordination and communication between the dev & QA team is the key…

#16 swapnil gujar on 12.03.13 at 7:36 am

Nice article!!! :)
However one doubt….BA will forward USER STORIES to developers , in the diagram it show BUGS –> DEV .can some please explain me .

#17 Aniket Deshpande on 12.09.13 at 11:47 am

Hi Swapnil,
Thanks for the feedback.
Developers will either fix bugs or implement new features (User Stories). That is what is depicted in the diagram. Essentially, anyone in the chain can raise a bug (not just QAs).

Leave a Comment