When a QA starts working on a mobile application project, the biggest question he/she faces is about the test environment.
With the growing market for mobile phones each day, it is a real challenge to decide the devices on which your mobile application testbed should comprise of. As it is not possible to purchase real devices beyond a limit and neither can every testing be done on emulators, the testbed has to be created in a smart way.
Android is an open source OS and has opened the doors for companies to launch their devices and that too at very low prices. Not everyone can buy an iPhone or a Samsung S7 or Nexus6 etc., hence people tend to buy affordable phones like Samsung S6 or Nexus 5 or Redmi Note4 etc.
What You Will Learn:
Types of Mobile Devices
On a very high level, the devices can be categorized as the High end and low-end devices:
The High-end devices are usually the ones with the latest Android OS, 4G or LTE enabled networks, 8 inches or more screen size, 7+GB memory, etc, and of course are very costly.
The Low-end devices are just the opposite ones with an older OS version, 2G or at the max 3G network, small screens, low memory, or phones with memory card slot not greater than 2 or 3 GB and hence are quite cheap (in other words affordable for everyone).
For Example, the following is the distribution of smartphones for 2017 based on the type of network:
[image source]
As you can see, there are only 28% of users are using the latest and the most advanced technology of LTE while a majority of the users are still on GSM.
Mobile Phone Testbed
As a Tester, what should my Mobile Phone Testbed be?
While selecting your testbed for mobile application testing, for manual or automation testing you can use emulators, simulators, and real devices.
Selection between emulators and real devices is a tough question because not all crashing issues are caught on emulators or simulators (the error popup or messages are often suppressed) while on the other hand buying all real devices can cost the company a lot.
I would suggest having one high-end phone, 1 or 2 mid-range phones, and at least 2 low-end phones.
The experience that I gained while working on a different variety of devices was that:
#1) High-End Phones: The apps run very smoothly on the high-end phones, obviously since you have 4G connections, 8GB memory, and a big screen the glitches or crashes will be the least. The high-end phones should be preferred for testing the functional aspects of the app but may not prove to be a good choice to test for scalability (against mobile data) or UI glitches or memory leaks.
#2) Mid-End Phones: The app runs smooth but still, scalability and memory leaks are caught like 1 out of 15 times. But this is not enough because crashing issues reported by customers for a mobile app is not acceptable and it leaves a bad image of the team, unlike desktop or web apps.
#3) Low-End Phones: The app may or may not run smoothly on a low-end phone because of certain factors.
Those factors include:
1) Slow Network: Currently low-end devices don’t have a network type beyond 3G and there are apps running in the background which use the network bandwidth. In a real scenario, it may be possible that the user has the Sync On for his/her email accounts, Whatsapp, Google Now, etc running in the background eating the bandwidth.
[image source]
2) UI: Low-end devices can have a screen size ranging from 4 inches to 6 or 7 inches thus scrolling down the list or tapping on icons is a little difficult. The icons are small in size and it is not easy to tap on them. Similarly scrolling down a long list may also cause glitches and either the cards in the list don’t load completely or they are distorted.
Following is an example of UI failure
3) Low Memory: Low-end devices do not have huge memory and most of the apps have around 5 – 6 GB at the max. Due to this, the apps run slow and at times the phone shows messages to close or ‘force stop’ some apps in order to free memory.
Why is Testing on Low-End Phones Important?
Depending upon your target audience, there is no need to use a low-end device every time but when you are doing the initial release of your app, you should definitely do full testing on a low-end device.
Across the world, users of the high-end costly phones are less and the users of mid and low-end phones are much more. If your app runs smoothly on a high-end phone, then it doesn’t mean that it will work absolutely fine on a low-end device too.
Out of my 3 years of experience, I would rather prefer testing on a low-end phone first.
Testing on a low-end phone is important because:
- The OS of the phone is a little old and the version is not the latest one.
- There are a lot of apps already installed on the phone using a huge memory, thus it is a perfect environment for testing ‘low memory’ tests.
- Interaction with the other apps for compatibility like using Google Maps, Phone Dialer, Messaging app, etc.
- Camera functionality is not advanced in low-end phones.
- If your app needs to send geo-coordinates, then the battery of a low-end phone drains out.
- Owing too-small screen, tapping on icons or scrolling, or navigating between screens can have issues, etc.
When I was testing my app on low-end devices, the following are the problems that I observed on just low-end phones and no other phones:
- Our app had a web service to send the geo-location every 10 minutes but as the low-end phone (that we were using) had 2G, the web service call was failing. Hence 2 major issues showed up – [i] the location wasn’t recording in the DB and [ii] the battery of the phone was draining out.
- While testing on 2G, we found that the web services were taking a little longer time to log in or fetch data for a screen or send data. Sometimes to see updated data it took even 15 minutes.
- We had cards to show the deliveries list for a day and while scrolling a long list of say 50 cards, the app crashed.
- While clicking the Back button, the navigation was failing and it was navigating 2 screens back rather than one screen.
- We had an image capture facility and after capturing the image we were re-sizing (reducing) the image. On the low-end phones, the camera crashed after taking 3-4 images in a go while re-sizing because the memory wasn’t getting freed from the previous image capture.
Following is an example of such an error that can come up:
- Some icons appeared too small to click and thus it blocked the testing of some functionality.
These are some of the most important failures which did not appear even once on the high-end device as per our team’s observation. All these issues were then fixed by the development team and we did a full regression on the low-end phone.
In fact, I remember that in the first release, we received a lot of camera crashing issues from the customers who were using low-end phones and after clicking around 20-30 images we could reproduce the issue. It was a tough challenge for us to work on as we were at the start of our app and the mobile application was completely a new one to us.
The delivery people were not using a high-end phone and only around 5% of them were having iPhone 5 or S3 (which was the latest at that time. While testing on a low-end device it would be great to use the mobile network rather than a high-speed Wifi.
After the first release we did our scalability, stress, and UI testing on 2 low-end phones and 1 mid-end phone, however, the issues were mostly found on the low-end phones. Just to be sure, we did a BVT on a high-end phone.
It is not necessary that you will come across the same kind of issues but you will surely observe issues with low-end phones which you will never come across on the high-end phones.
Another example that comes to my mind is that a lot of apps use Google Maps to load locations or show their location.
On a low-end phone, the version of Google Maps may be old and hence there may be compatibility issues when your app makes a call to load a location in Google Maps or due to a slow network, it may take a longer time to load. It may also be possible that the Maps may not load the exact location which your app actually wanted.
Emulator or Simulator as a Replacement for a Low-End Phone
Can I use an Emulator or Simulator as a replacement for a Low-End Phone?
It is not advisable to use an emulator or simulator for low-end phones because the development team of my app tried that and they were doing their BVT on the emulator hence they missed a few crashing issues.
Initially, when we started working on a mobile app, it was new to us and the development team wasn’t aware that a lot of error messages are suppressed by the emulators and the errors were not shown. We found the issues while testing on real devices.
After that, even our development team started using the real device for their testing rather than an emulator or simulator. Another reason to avoid using emulators is that it is not possible to create a 2G or 3G network and they use the high-speed network of your laptop or desktops. However, you can do UI and navigation testing on an emulator.
Testing Strategy for Low-End Phones
As a QA, what should be my ‘testing’ strategy for low-end phones?
To decide the testing strategy for low-end phones be very clear about your app requirements and what features of a low-end phone will affect your app and its performance. The defects found on low-end phones are very hard to find and tricky as well and more than that for a QA it is very difficult to reproduce, verify and regress such bugs.
Following are some pointers to decide on your strategy, these helped me and my team, hope they will help you as well:
- Never do a single hand testing on a low-end phone, the more the number of minds involved the more are the chances of finding hidden bugs. Hence try ‘QA pairing’ or testing in a circle like if QA1 Tests functionality F1, then to have QA2 verify the bugs reported for F1, have QA3 do the regression testing.
- Discuss your strategy, and your findings with the other QAs of your team, this helps a lot because maybe they see something that you may not or may have some experience that can help you.
- Until your app doesn’t stabilize, have a separate testing task created for low-end phones and keep them estimated.
- It is practically not possible to create an ideal test environment and also not to have a number of phones. Hence (with the agreement of your Manager or SM) check with the other colleagues in your company who have such devices and have them install and use your app.
- The same is for Beta testing as well, have your BAs or Product Owner use a low-end phone and do one round of end to end-run.
- When the major bugs are found (like the ones mentioned above), do sit with the assigned developer while they fix and also understand from them the reason behind the bug. This will help you not only in verification but also in regression. You can also take opinions from the developers about how to regress.
- Last but not least (in my case, that was the first mobile app project) document such bugs because it is a big knowledge that you gained and can help others as well. Document how the bug surfaced, and how was it fixed, verified, and regressed.
Conclusion
The mobile application is a fast-growing market and there is a new OS phone launched every other month, hence being a QA it is not possible to test for every OS-phone combination and it is not feasible too. Therefore while choosing your testbed, make a smart choice and select the phones, emulators, etc for each category of a high, medium, and low end.
Testing on a different category may vary according to your app requirement but as per my experience, there’s no harm in testing (end to end) on a low-end device. The reason is the outdated features, slow network, etc resulting in issues that you may not observe on high-end phones.
In Mobile application testing, crashing issues are not always found testing, and sometimes customers report issues that you never noticed. Hence your Product Owner may once or twice excuse missed issues but not every other time, so being a QA try to test more on low-end phones.
If your Managers or BAs are not interested in having low-end devices as a testbed, try to convince them and be firm, unless they have strong reasons or surety that the target customers are all using the best phones.
To know more about the importance of Field Testing, take a look at our upcoming tutorial.