How to Implement Efficient Test Automation in the Agile World

Automation in Agile is very critical.

Think about the many features that are added and delivered in every Sprint.  There has to be a way to make sure that the newly added feature is not impacting the existing functionality.

Because of the low Sprint duration, it is practically impossible to execute the entire suit every time the product is incremented at the Sprint end.  Having an automated test suit would definitely play a bigger role here.

However, introducing and maturing into automation would definitely take some time. Doing an initial investment in planning and designing the automation activity would definitely pay off in the long run.

Automation in Agile

In this 3rd part of Agile Testing advanced series, I am trying to cite a few pointers to consider based on my experience, as you bring automation to your project.

Also, read part 1 and part 2 first to have the better understanding of the topic.

What to Automate in Agile?

Whenever we are planning to introduce automation into our projects, most of us immediately vote for either the “Smoke Tests suit” or the “regression test suit” to be the best candidate for automation. Of course, they are, but when we think of the automation test pyramid, we can conclude that it’s just the top layer of the pyramid that we are talking about.

Apart from above layer we still have the service layer and the unit layer that are more important.

So what tests, other than Smoke tests and Regression tests, can be good candidates for automation?

#1) Builds and Deployments

In traditional environments, we have predefined builds that can be weekly, fortnightly or sometimes even monthly. One of the reasons is that these deployments take time. The problem with this approach is that we have to wait for the predefined dates to get the bugs fixed or to get the new features implemented, so there is a delay.

The second reason was – by the time testers finish up with the testing and come up with bugs and defects, the programmers have moved on to different pieces of implementation and have less interest in resolving the bugs of the older application. This approach also delays the time for making the feature available in production.

Building and deployments are the entities that are repetitive and sometimes boring. It can also take hours to deploy a build, which delays the testing and eventually the feedback. Being a repetitive task, deployments becomes a good candidate for automation.

Also read => The Release and Deployment Management Process

A few of the advantages of having automated build deployment are:

  • No chance of making deployment mistakes (human errors like copying the incorrect file or copying file to the incorrect location can be avoided)
  • Bug/features are available to test as soon as they are fixed
  • Testers get more time to test
  • The feature is ready to be moved to production in less time
  • Quick feedback

#2) Unit tests/Component tests

I have already talked about the importance of automating the unit layer by using the TDD approach in my last tutorial.

This forms the lowest layer of the pyramid, hence the foundation and any foundation needs to be rock solid. The development team should collaborate and work together to accommodate most of the test into this layer.

#3) API/Web Service Testing

Web services are the medium in which two applications exchange the data or information in terms of request and response, without bothering with the underlying architecture or the technology. In more simple terms – giving a request and validating the response is what we normally do in the web services testing.

Testing the web services entails writing programs to call those web service methods and validating the value/s that it returns. We can even test the services for various permutations and combinations. Have all the test data in the excel sheet and your program can read the data and call the testable service by passing the test data as a parameter and validate the results.

This particular testing is part of the middle layer of the pyramid. Most of the functional testing can be pushed into this layer. Resolving defects that arise in this layer becomes easy to fix and they are not postponed until the UI is available.

#4) Testing behind the GUI

Automating the testing behind the GUI is comparatively easier than automating the actual GUI. Another advantage is that irrespective of the UI changes, functionality remains intact. Even if some of the UI element is changed, the functionality of the feature does not change. This technique mainly focuses on the business logic and rules.

The test cases are mostly written in a tabular format or in a spreadsheet and fixtures/code snippets are written that accept the input from these tables and returns the results. The results are generated immediately and provide a great platform for the non-technical stakeholders to run these tests and get the expected results. One of the tools used for achieving this technique is Fitnesse.

#5) Non-functional testing

This non-functional testing technique basically involves the Load, Performance and Stress testing. There are various tools readily available in the market that can be utilized to automate these tests.

#6) Data Comparisons

Many of our testings require us to compare data files, including text files, CSV or excel files

  • These files can be compared against baselines for doing data validation
  • Comparisons can be of same data but of a different format. This basically happens when we have two of the same files generated from two different sources

These comparisons can be repetitive, therefore automated.

#7) Searching

Searching for a specific entity from a large bunch of files can also be tedious and God helps us if that is a repetitive task. One example is searching through log files. If this is also a tedious and repetitive task than we should think about automating it.

#8) Repetitive Tasks

Any task starting with interacting with end users or writing stories to developing it, if it is repetitive, should be considered in automation. We should understand that doing automation does not mean that there has to be a sophisticated tool/technology involved in it. It can be a simple VB macro or a Java program with a Javascript to solve the purpose.

Where to Start?

There are no bullet points or a step by step guide that says where to start automation. Kicking off automation for the team requires you to brainstorm and apply deep thoughts on what aspects you are looking to automate, or what is the ultimate goal of the automation?

You can start by:

  • Identifying the repetitive tasks,
  • Identifying the pain areas of the application
  • Identifying the testing challenges

If you have no automation in the tour project/team, then you can probably go for a multi-layered approach where the unit tests can be targeted first to automate. This would give you the highest ROI.


Simultaneously, testers can start working on smoke test suit and then the regression. Once the team has gained the skills and feels comfortable, gradually move towards automating the other repetitive tasks.

Do not directly jump into buying a new tool without evaluating your needs. As I said earlier, a simple program or a macro can solve your purpose of automating some of the repetitive tasks. So, before deciding to buy a tool, do a POC and evaluate whether that tool would be effective to use.

Please go through these documents where I have provided more details on how to select correct test cases for automation and some insights on Estimating automation efforts in the following articles manual to automation testing process challenges and test estimation of selenium automation project.

Once the scope of automation and tool is finalized, next is to design the framework.

Remember, in Agile, the framework is evolved. Do NOT target designing the entire framework first and then implement. Design and implement for the MVP (Minimum Viable Product) and then enhance the existing framework to include more features. You also need to apply good coding and development practice if you want your automation suite to be robust.

Some Best Practices

  • Don’t target to Automate 100% at one go. Start small. Remember it is an evolving process
  • Follow the same Agile practices that you follow for any software development. Automation also requires proper planning and designing. You would not want to increase your technical debts when you are automating
  • Create your test automation backlog. This backlog can range from implementing a new feature to enhancing an existing feature. Give story points to your identified items and assign it accordingly. Take these backlog items to your Sprint and track it using a Kanban board
  • Write the acceptance criteria for your automation stories. These acceptance criteria can include:
    • Integration of the test suite with CI
    • Porting the suit to a centralized location
    • Send the results via email
    • Provision of sending the error log files when the test fails
    • Any other criteria….
  • Do not overspend time evaluating a new tool. You can create a prioritized checklist of what all you want from the new tool and decide a timeline for evaluating it. If you do not see your results in the stipulated time, move on to the next one
  • Make a judicious decision on what to automate. Not every piece of automation is effective and yields a positive ROI. Do not automate just for the sake of automation
  • Make use of the proper development environment. Do not keep the code to your local. Have a repository to keep your code and make a habit of checking on your code at the end of the day
  • In a similar fashion, try to execute your automated tests from a centralized location. Make it person independent. It should be that anybody from the team can trigger the scripts from their machine and results are obtained via email

What are Agile principles that can be applied to automation?

Some very simple tips:

  • Keep things simple. Do what is needed. I have seen many instances where we deliver sugar coated implementation which makes the automation unnecessarily complicated. Let’s avoid the things which are not needed
  • Doing simple things does not mean doing the easiest things. This means taking baby steps to achieve your automation goals. You may take up a simple feature to automate, but it may happen that the implementation of automation turns out to be a complex one
  • Apply the whole team approach. I believe everybody is a tester in an agile team. Let’s not restrict the automation job either only with the testers or only with the developers. Each of the disciplines has to step into each other’s shoes to achieve automation for the project. This approach would also be effective to resolve any of the technical issues that come with implementation
  • The framework is evolved in Agile. Do not try to provide too many features that may unnecessarily make the piece of automation complex
  • Take the time to do it right. Take some time to design it properly to avoid the technical debts
  • Get frequent feedback
  • Apply proper coding standards and practice. Design should be simple, apply the OPPS concepts and try to keep the tests independent of each other. Consider factors like “maintainability” of the test suit

Do I see any challenges while automating in Agile?

Automating in the Agile world does come with its own challenges:

  • We need to plan really well. Deciding the appropriate test suite, tool, framework, and approach, all need a proper strategy. However, we should remember NOT to over plan. Keep the MVP (Minimal Viable Product) in mind
  • Compromising on the quality of the code because we want to deliver fast: We must remember that technical debts hold well in automation too
  • Team most of the time teams do not follow the “Whole-Team-Approach” and leave the entire responsibility of coding and maintaining the automated suite for the testers which adds to the responsibility of testers
  • Automating the functional tests is tougher than automating the UI

Amongst all these challenges, the most critical challenge is to upgrade the skills of testers.

Doing and maintaining the automation for a team is almost like a programming (development) activity which the programmers (developers) do.  Not only just the implementation but also integrating the automated suit to CI is important and requires that testers learn and adopt new skills and learn new tools and technologies.

Some Open Source Tools that fits into Agile

  • Selenium WebDriver – For UI
  • Selenium Grid – For parallel execution
  • Cucumber – For BDD
  • JMeter – For performance testing
  • SoapUI – For web services
  • WireMock – Web service testing when the webs service is not available.
  • Appium – For Mobile

Let me conclude with the famous Agile test quadrants:

Agile test quadrants

Quadrant 1 is the Unit and the components test that can be automated with the TDD approach.

Quadrant 2 talks about the functionality testing, where we can apply the BDD approach.

Quadrant 3 is the only quadrant which has a scope of manual testing.

Quadrant 4 basically talks about the testing that can be achieved by some tools.  This takes care of the Load tests, Stress tests, Volume tests and Security tests.

Conclusion

There is a lot of scope of automation apart from the Smoke tests and Regression tests. Therefore, we have to break free from the concept of confining automation only to these types of testing, which in turn means that the skill set of a tester in Agile demands more than just finding bugs and defects.

Testers need to be more collaborative and sharpen their programming/automation skills. If more and more tests are automated, it would give the testers more time to engage themselves in more sophisticated and challenging tasks.

About the author: This article is by STH team member Shilpa. She is working in the software testing field for the past 10+ years in domains like Internet advertising, Investment Banking, and Telecom.

Please share your comments and thoughts below.




Recommended reading

3 comments ↓

#1 Krunal

A genuine and fruitful explanation. Nice work!

#2 Devraj Valecha

Great article!

#3 Nayan

Great post, thanks for sharing. But here I want to introduce Kostacre Canada Corporation. These guys are really doing a great job in Software Testing. Kostcare is a trustworthy Software testing company having a very experienced team of the tester.

Leave a Comment