Field Testing – Mobile Applications:
Often while testing our mobile application, a question comes to our mind is that ‘whether automation testing, regression, load, web service testing are enough for our app or should we test anything more?’
In this tutorial, I will be providing a deep insight into Field testing, its importance, strategy and explain when should it be done as well.
Well, it may be enough or it may not be because all these tests like automation, web service, load etc., come under the ‘In-House’ testing category. But if you want to implicate the end user role, there is yet one more type of test that needs to be done and that is nothing but ‘Field Testing’.
As the name itself suggests, it involves going out of your strong Wifi zone and using your 2G or 3G or 4G network to check the usability of your application. You may try using the app while driving your car or while taking a walk or sitting at your home according to your comfort.
For basic information on Mobile application testing, you can refer the following tutorials.
- Beginners guide to mobile application testing
- 5 mobile testing challenges and solutions
- Why mobile testing is tough
Table of Contents:
Classification of Mobile Application Testing
If you are testing a web application on a mobile, the types of the test remain the same as you do for your laptops or computers like functional, UI, boundary, load, etc. But for testing a mobile application, whether Hybrid or Native, testing is classified into two categories at the very initial level.
They are:
- In-House Testing
- Field Testing
The base for the difference between these two categories is the role that you are impersonating i.e. Are you testing as a QA to verify the app against the requirement or are you using the app as an end user to fulfill your need?
Given below are few differences between both:
| In-House Testing | Field Testing |
|---|---|
| This testing is done right from the initial till release phase to verify that the application is meeting its stated requirements in terms of functionality, scalability, usability etc. This is done against the user stories or the project requirement documents provided by the customer. | This testing is done in the last phase when the regression is done for the application and the application is called stable by the ‘Team’ before the release. There may or may not be a requirement given for this from the customer. |
| The type of testing includes functional, regression, UI, performance, automation. | The type of testing mainly include the functional and usability of the application. |
| This testing is done on both WiFi and mobile networks. | This is strictly done on Mobile Networks. |
| Test labs are created in a room or on a QA’s desk. | QA’s need to step out and test while walking around or at home or driving. |
| Testing can be done on emulator or simulator. | Testing is done only on real devices. |
Importance of iOS and Android Field Testing
As explained above, Field testing is done to verify the behavior or usability of your application as an end user. This is basically done on your phone’s mobile network by walking around on the streets or a market, by driving a car or just being at your home.
In today’s world, we use our mobile apps everywhere, anytime, being at home doing shopping/booking tickets/checking emails etc., or standing in a market searching for nearby places/restaurants/malls or be it driving to a location by searching on maps.
Mobile applications have now become a part of our daily life.
Following are some examples which explain the need for testing your mobile app in-field and not just in-house:
- It is not necessary that every time you will have a strong Wifi or hotspot around, especially in India. Hence you have to rely on your phone’s mobile network.
- There are places where we face coverage issues and the phone range will be medium or low. This may be due to less mobile towers or densely covered with trees. At my place, I face this problem a lot in my drawing and dining room which is frustrating all the times.
- It may be possible that you are driving your car and you need to access your emails or get direction to some place which is unknown to you.
- When we go out on a vacation, we do not have (every single) information about the city or the place. When I travel with my family on vacation, we always go on our own and use the apps to find about the city, be it restaurants, places to visit etc.
- Today we prefer online transfers by using apps like PayTm, PayZApp etc., rather than using cash always.
- Using chat apps, YouTube etc., anywhere.
These are some of the real-time examples that define the usage of mobile apps. Hence it becomes important to test your app in-field. It may be possible that your target audiences are in a well-developed country having hot-spots or Wifi available everywhere.
But still, you should do at least one field testing of your app because:
1) When you test your app while driving a car or walking or sitting in a place with less range, you get to know if your app is having any crashing issues while sending or fetching data.
We had the app login itself crashing for the very first release of our app, we tested on a phone which had 3G. Still, the login got timed out and crashed.
2) Users hate it when they see a loading symbol appearing for more than 5 minutes. It is very important to verify what is the response time taken by your server, web service and your app to load the information.
These are the two critical pointers that are verified in a field test and this also helps you to be assured that your app is really stable.
How is Field Test Affected?
Is my field test affected by the Hybrid or Native way?
Before answering this question, let me first give you a basic explanation of Hybrid and Native apps.
Hybrid Apps:
These are basically the web apps which are packaged into the native wrapper and are mostly used to develop cross-platform applications (to keep the code bases same for Android and iOS). They look like Native apps but in actual the information is loaded as the user navigates through pages.
Native Apps:
These are specifically developed for an OS using the features of the OS to its best. All the information is loaded in one go when the app is started.
I have worked on both Hybrid and Native app testing and as a QA, testing for both was important for my app. There is a difference between Hybrid and Native app that unlike Native apps, Hybrid apps don’t directly use the features of your phone like GPS, Location etc. The hybrid apps use the phone’s or OS’s features via plug-ins.
I strongly feel that for every major release for a Native app, Field test should be done because it’s using your phone’s and OS’s features.
Performing a Field test will indicate the performance of your app with respect to the same. It will indicate whether the phone is a high end or low-end one, how is the performance, stability, and usability of your application.
Another reason to do a Field test for every major release is that native apps need to be updated with new versions of the OS. Hence compared to the Hybrid apps, versions of Native are more or if a new OS version is launched a field test is needed even if there are no changes done in the app’s features.
Strategy for the Field Test
What should be my strategy for the Field Test?
For Field testing, the prime and important step to be understood is the impact of the network or data on the major functionalities of your app. Hence have a thorough discussion with the development team about the functionalities.
Following are the pointers to decide the strategy for field testing:
1) The prime step as a QA is that tag your test cases with ‘Field Test’ and create a suite for the same. Discuss with your BA and QA team about the Suite and see if they have anything to add to it.
2) Depending on the purpose of your app and the users, create some separate test cases for how, when and where can your app be used. You may not need to execute these for in-house testing.
3) Go for a field test only when the app passes the regression test and when there are no functional issues open.
4) For the very first release create a document (chart) with all the details of test cases, the person executing, an area where it was tested and the bugs reported. Once created you can use this for future releases.
5) Use a High end and a Low-end device for field testing and try to keep the same OS, this will help to do an analysis of the bugs found. At times we see bugs in field test but not in the in-house testing.
6) Create a team of 4-5 people and have them do the field test in different areas and for sure try to use the app while driving and a place with a low data range. You can also have your manager, BA involved in the field test.
7) Tag the bugs found with a tag like ‘Field Test’, it will be easy for you to refer to future releases.
8) If your app is sending geolocation, test the functionality thoroughly and I would suggest executing all the test cases related to the same.
It is not necessary that you will find bugs or defects and the test may run smooth. As per my experience, keep your focus on the performance and the time it is taking for responding. As mentioned earlier a ‘Loading’ screen will discourage people from using your app.
If your app is a commercial app like Zomato, Jabong, BookMyShow etc., then try to do a field test using a team of 20-25 people because it will indicate the app’s performance when 20-25 users will hit the app.
When to do a Field Test for an App?
It is not necessary that all apps need to be field tested but I always personally recommend to do a field test since it shows the app’s performance in the real world scenario. You can discuss this with your BA, scrum master or manager, and the product owner if they are not aware or aren’t interested in doing. As a QA, give your points about the need for a field test.
A field test should be done only after regression testing of the app is done, the functionalities for a release are implemented, bugs are verified, fixed and most importantly the app is stable.
Important factors of Field Test include:
- First Release of the app: It gives a great advantage to do an in-depth field test of the app for the first most release. As the app is yet to go to the Play Store, it helps you and your team to analyze the performance by using the app like a normal user.
- An app is Native: As mentioned before, the field test is a must for every release of a native app as they use the features of the phone and the OS.
- When new OS version is launched: When a new OS version is launched, you need to verify if the app is stable, compatible and performs well against the new OS.
- Any major release for new or changed features: When new features are introduced or existing features are updated, it is good to do a field test to make sure that the performance of the app is not affected.
- Field Testing as a part of your routine test: Field testing for a mobile app should be treated just as a routine test like regression, automation and should never be ignored. Even if your app is stable or if you have been working on it for more than 1 or 2 years do a field test every time when a major release happens.
Examples of Tests that can be Verified in a Field Test
Let us consider the app of Zomato which we use to search for restaurants around us or in a specific area.
Given below are few examples of test cases that can be taken up for a field test:
#1) The basic and the very first step would be to log in that I would do as a customer.
Here note that first of all, it should not take more than a minute to log in with the Facebook or Google account that I choose, secondly the moment I am logged in it should ask me for a permission to turn on my phone GPS (if it’s off) else should load the restaurants list for my detected location.
Above mentioned are the 2 important verification to be done in the very first test.
#2) Verify that when you change the location, the restaurant list gets updated, try this test while walking out on the road. The list should update and the cards of the restaurants should be shown with pictures without any glitches.
#3) Verify the above test case also while driving a car.
#4) Verify (while walking on the road) the end to end flow of the app i.e. right from logging in, searching for a location and ordering food from a restaurant. Your order details should be sent to the restaurant successfully.
#5) Execute the same above test case while driving a car.
#6) Verify on a 3G (by changing your data mode to 3G) for an end to end flow for ‘booking a table’ i.e. again change the location, select a restaurant from the list and book a table.
#7) If you have a coupon, check if you can use it successfully while ordering food from Zomato.
#8) Verify if you can successfully write a feedback for the restaurant (while walking) and if the feedback is sent to the restaurant for review or at least if it shows up in the reviews list of the restaurant.
These are some of the examples of test cases while defining the basic use of the app and should be tested in the field test. Here you can also create a matrix for Phone model (High end or low end) Vs the type of network (2G, 3G etc) and execute the test cases for the same.
Conclusion
Any mobile application can be called stable and fast not just by passing the regression test, bug verification, stress or load test but only when it successfully passes a field test as well.
Field test should be done for every release to the possible extent. For mobile applications, the field test is a part of the testing cycle and should never be avoided or taken lightly.
Field testing plays a very important role in defining your apps’ performance for the very first release. If your team isn’t aware of this, explain them about it and the need for it.
In our next tutorial, we will discuss more on “Testing the latest phone or latest OS“.