Agile Test Automation Strategy: What and How to Automate

Quickly accelerate your Agile journey with smart test automation. Unleash the full potential of your development cycle with efficient test automation techniques in the agile world:

Automation in Agile is very critical. Think about the many features that are added and delivered to every Sprint.  There has to be a way to make sure that the newly added feature is not impacting the existing functionality.

Due to the low Sprint duration, it is practically impossible to execute the entire suit every time the product is incremented at the Sprint ends.  Having an automated test suit would play a bigger role here.

However, introducing and maturing into automation would take some time. Doing an initial investment in planning and designing the automation activity would pay off in the long run.

Test Automation in Agile World

In this 3rd part of the Agile Testing advanced series, I am trying to cite a few pointers to consider based on my experience, as you bring automation to your project.

Also, make sure to read part 1 and part 2 first to get a better understanding of the topic.

What Should Be Automated in Agile?

Whenever we are planning to introduce automation into our projects, most of us immediately vote for either the “Smoke Tests suit” or the “regression test suit” to be the best candidate for automation. Of course, they are, but when we think of the automation test pyramid, we can conclude that it’s just the top layer of the pyramid that we are talking about.

Apart from the above layer we still have the service layer and the unit layer that are more important.

So what tests, other than Smoke and Regression tests, can be good candidates for automation?

#1) Builds and Deployments

In traditional environments, we have predefined builds that can be weekly, fortnightly, or sometimes even monthly. One of the reasons is that these deployments take time. The problem with this approach is that we have to wait for the predefined dates to get the bugs fixed or to get the new features implemented, so there is a delay.

The second reason was – that by the time testers finish up with the testing and come up with bugs and defects, the programmers have moved on to different pieces of implementation and have less interest in resolving the bugs of the older application. This approach also delays the time it takes to make the feature available in production.

Building and deployments are entities that are repetitive and sometimes boring. It can also take hours to deploy a build, which delays testing and eventually feedback. Being a repetitive task, deployments become a good candidate for automation.

Also read => The Release and Deployment Management Process

A few of the advantages of having an automated build deployment are:

• No chance of making deployment mistakes (human errors like copying the incorrect file or copying a file to the incorrect location can be avoided)
• Bugs/features will be available to test as soon as they are fixed
• Testers get more time to test
• The feature is ready to be moved to production in less time
• Quick feedback

#2) Unit Tests/Component Tests

I have already talked about the importance of automating the unit layer by using the TDD approach in my last tutorial.

This forms the lowest layer of the pyramid, hence the foundation and any foundation needs to be rock solid. The development team should collaborate and work together to accommodate most of the tests into this layer.

#3) API/Web Service Testing

Web services are the medium in which two applications exchange data or information in terms of request and response, without bothering with the underlying architecture or the technology. In more simple terms – giving a request and validating the response is what we normally do in web services testing.

Testing the web service entails writing programs to address those web service methods and validating the value/s that it returns. We can even test the services for various permutations and combinations. Have all the test data in the Excel sheet and your program can read the data and call the testable service bypassing the test data as a parameter and validating the results.

This particular testing is part of the middle layer of the pyramid. Most of the functional testing can be pushed into this layer. Resolving defects that arise in this layer becomes easy to fix and they are not postponed until the UI is available.

#4) Testing Behind the GUI

Automating the testing behind the GUI is comparatively easier than automating the actual GUI. Another advantage is that irrespective of the UI changes, functionality remains intact. Even if some of the UI elements are changed, the functionality of the feature does not change. This technique mainly focuses on business logic and rules.

The test cases are mostly written in a tabular format or in a spreadsheet and fixtures/code snippets are written that accept the input from these tables and return the results. The results are generated immediately and provide a great platform for the non-technical stakeholders to run these tests and get the expected results. One of the tools used for achieving this technique is Fitnesse.

#5) Non-Functional Testing

This non-functional testing technique basically involves Loading, Performance, and Stress testing. There are various tools readily available in the market that can be utilized to automate these tests.

#6) Data Comparisons

Many of our tests require us to compare data files, including text files, CSV or Excel files

• These files can be compared against baselines for doing data validation
• Comparisons can be of the same data but in a different format. This happens when we have two of the same files generated from two different sources

These comparisons can be repetitive and therefore automated.

#7) Searching

Searching for a specific entity from a large bunch of files can also be tedious and God helps us if that is a repetitive task. One example is searching through log files. If this is also a tedious and repetitive task then we should think about automating it.

#8) Repetitive Tasks

Any task starting with interacting with end-users or writing stories to develop it, if it is repetitive, should be considered in automation. We should understand that doing automation does not mean that there has to be a sophisticated tool/technology involved in it. It can be a simple VB macro or a Java program with Javascript to solve the purpose.

Where to Start?

There are no bullet points or a step-by-step guide that says where to start automation. Kicking off automation for the team requires you to brainstorm and apply deep thoughts on what aspects you are looking to automate, or what is the ultimate goal of the automation.

You can start by:

• Identify repetitive tasks,
• Identifying the pain areas of the application
• Identifying the testing challenges

If there is no automation in the tour project/team, then you can probably go for a multi-layered approach where the unit tests can be targeted first to automate. This will give you the highest ROI.

Simultaneously, testers can start working on the smoke test suit and then the regression. Once the team has gained the skills and feels comfortable, gradually move towards automating the other repetitive tasks.

Do not directly jump into buying a new tool without evaluating your needs. As I said earlier, a simple program or a macro can solve your purpose of automating some of the repetitive tasks. So, before deciding to buy a tool, do a POC and evaluate whether that tool would be effective to use.

Please go through these documents where I have provided more details on how to select the correct test cases for automation and some insights on Estimating automation efforts in the following articles manual to automation testing process challenges and test estimation of a selenium automation project.

Once the scope of automation and tool is finalized, the next step is to design the framework.

Remember, in Agile, the framework has evolved. Do NOT target designing the entire framework first and then implementing it. Design and implement the MVP (Minimum Viable Product) and then enhance the existing framework to include more features. You also need to apply good coding and development practice if you want your automation suite to be robust.

Some Best Practices

• Don’t target to Automate 100% at one go. Start small. Remember this is an evolving process
• Follow the same Agile practices that you follow for any software development. Automation also requires proper planning and designing. You do not want to increase your technical debts when you are automating
• Create your own test automation backlog. This backlog can range from implementing a new feature to enhancing an existing feature. Give story points to your identified items and assign them accordingly. Bring these backlog items to your Sprint and track them using a Kanban board
• Write the acceptance criteria for your automation stories. These acceptance criteria can include:
  • Integration of the test suite with CI
  • Porting the suit to a centralized location
  • Sending the results via email
  • Provision to send the error log files when the test fails
  • Any other criteria….
• Do not overspend time evaluating a new tool. You can create a prioritized checklist of what you want from the new tool and decide on a timeline for evaluating it. If you do not see your results in the stipulated time, move on to the next one
• Make a judicious decision on what to automate. Not every piece of automation is effective and yields a positive ROI. Do not automate just for the sake of automation
• Make use of the proper development environment. Do not keep the code to your local. Have a repository to keep your code and make a habit of checking on your code at the end of the day
• Similarly, try to execute your automated tests from a centralized location. Make that person independent. It should be that anybody from the team can trigger the scripts from their machine and results can be obtained via email

What are Agile Principles That Can be Applied to Automation?

Some very simple tips:

• Keep things simple. Do what is needed. I have seen many instances where we have delivered sugar-coated implementation which makes automation unnecessarily complicated. Let’s avoid things that are not needed
• Doing simple things does not mean doing the easiest things. This means taking baby steps to achieve your automation goals. You may take up a simple feature to automate, but the implementation of automation can turn out to be a complex one
• Apply the whole team approach. I believe everybody is a tester in an agile team. Let’s not restrict automation jobs either only with testers or only with developers. Each of the disciplines has to step into each other’s shoes to achieve automation for the project. This approach would also be effective in resolving any of the technical issues that come with implementation
• The framework has evolved in Agile. Do not try to provide too many features that may unnecessarily make the piece of automation complex
• Take the time to do it right. Take some time to design it properly to avoid technical debts
• Get frequent feedback
• Apply proper coding standards and practice. Design should be simple, apply OOPS concepts, and try to keep the tests independent of each other. Consider factors like “maintainability” of the test suit

Do I See any Challenges While Automating in Agile?

Automating in the Agile world does come with its own challenges:

• We need to plan really well. Deciding the appropriate test suite, tools, frameworks, and approaches all need a proper strategy. However, we should remember NOT to over plan. Keep the MVP (Minimal Viable Product) in mind
• Compromise on the quality of the code because we want to deliver fast: We must remember that technical debts hold well in automation too
• Teams, most of the time teams, do not follow the “Whole-Team-Approach” and leave the entire responsibility of coding and maintaining the automated suite to the testers which adds to the responsibility of testers.
• Automating functional tests is tougher than automating the UI

Amongst all these challenges, the most critical challenge is to upgrade the skills of testers.

Doing and maintaining automation for a team is almost like a programming (development) activity that the programmers (developers) do.  Not only just the implementation but also integrating the automated suite to CI is important and requires that testers learn and adopt new skills and learn new tools and technologies.

Some Open Source Tools that Fit into Agile

• Selenium WebDriver: For UI
• Selenium Grid: For parallel execution
• Cucumber: For BDD
• JMeter: For performance testing
• SoapUI: For web services
• WireMock: Web service testing when the web service is not available.
• Appium: For Mobile

Let me conclude with the famous Agile test quadrants:

Quadrant 1 is the Unit and the components test that can be automated with the TDD approach.

Quadrant 2 talks about functionality testing, where we can apply the BDD approach.

Quadrant 3 is the only quadrant that has a scope of manual testing.

Quadrant 4 talks about the testing that can be achieved by some tools.  This takes care of Load tests, Stress tests, Volume tests, and Security tests.

Conclusion

There is a lot of scope for automation apart from the Smoke and Regression tests. Therefore, we have to break free from the concept of confining automation only to these types of testing, which in turn means that the skill set of a tester in Agile demands more than just finding bugs and defects.

Testers need to be more collaborative and sharpen their programming/automation skills. If more and more tests are automated, it would give the testers more time to engage themselves in more sophisticated and challenging tasks.

About the author: This article was written by STH team member Shilpa. She has been working in the software testing field for the past 10+ years in domains like Internet advertising, Investment Banking, and Telecom.

Please don’t forget to share your comments and thoughts below.