Why Mobile App Testing on Low-end Devices is Important?

When a QA starts working on a mobile application project, the biggest question he/she faces is about the test environment.

With the growing market for mobile phones each day, it is a real challenge to decide the devices on which your mobile application testbed should comprise. As it is not possible to purchase real devices beyond a limit and neither can every testing be done on emulators, the testbed has to be created smartly.

Android is an open-source OS and has opened the doors for companies to launch their devices and that too at very low prices. Not everyone can buy an iPhone or a Samsung S7 or Nexus6 etc., hence people tend to buy affordable phones like Samsung S6 or Nexus 5 or Redmi Note4, etc.

Types of Mobile Devices

On a very high level, the devices can be categorized as the High end and low-end devices:

The High-end devices are usually the ones with the latest Android OS, 4G or LTE enabled networks, 8 inches or more screen size, 7+GB memory, etc, and of course, are very costly.

The Low-end devices are just the opposite ones with an older OS version, 2G or at the max 3G network, small screens, low memory, or phones with memory card slot not greater than 2 or 3 GB and hence are quite cheap (in other words affordable for everyone).

For Example, the following is the distribution of smartphones for 2017 based on the type of network:

As you can see, there are only 28% of users are using the latest and the most advanced technology of LTE while a majority of the users are still on GSM.

Mobile Phone Testbed

As a Tester, what should my Mobile Phone Testbed be?

While selecting your testbed for mobile application testing, for manual or automation testing you can use emulators, simulators, and real devices.

Selection between emulators and real devices is a tough question because not all crashing issues are caught on emulators or simulators (the error popup or messages are often suppressed) while on the other hand buying all real devices can cost the company a lot.

I would suggest having one high-end phone, 1 or 2 mid-range phones, and at least 2 low-end phones.

The experience that I gained while working on a different variety of devices was that:

#1) High-End Phones: The apps run very smoothly on high-end phones, obviously since you have 4G connections, 8GB memory, and a big screen the glitches or crashes will be the least. The High-end phones should be preferred for testing the functional aspects of the app but may not prove to be a good choice to test for scalability (against mobile data) UI glitches or memory leaks.

#2) Mid-End Phones: The app runs smoothly but still, scalability and memory leaks are caught 1 out of 15 times. But this is not enough because crashing issues reported by customers for a mobile app are not acceptable and it leaves a bad image of the team, unlike desktop or web apps.

Those factors include:

1) Slow Network: Currently low-end devices don’t have a network type beyond 3G and apps are running in the background which use the network bandwidth. In a real scenario, it may be possible that the user has Sync On for his/her email accounts, WhatsApp, Google Now, etc running in the background eating the bandwidth.

[image source]

2) UI: Low-end devices can have a screen size ranging from 4 inches to 6 or 7 inches thus scrolling down the list or tapping on icons is a little difficult. The icons are small in size and it is not easy to tap on them. Similarly scrolling down a long list may also cause glitches and either the cards in the list don’t load completely or they are distorted.

Following is an example of UI failure

3) Low Memory: Low-end devices do not have huge memory and most of the apps have around 5 – 6 GB at the max. Due to this, the apps run slow, and at times the phone shows messages to close or ‘force stop’ some apps to free memory.

Why is Testing on Low-End Phones Important?

Depending on your target audience, there is no need to use a low-end device every time but when you are doing the initial release of your app, you should do full testing on a low-end device.

Across the world, users of the high-end costly phones are less and the users of mid and low-end phones are much more. If your app runs smoothly on a high-end phone, then it doesn’t mean that it will work fine on a low-end device too.

Out of my 3 years of experience, I would rather prefer testing on a low-end phone first.

Testing on a low-end phone is important because:

• The OS of the phone is a little old and the version is not the latest one.
• There are a lot of apps already installed on the phone using a huge memory, thus it is a perfect environment for testing ‘low memory’ tests.
• Interaction with the other apps for compatibility like using Google Maps, Phone Dialer, Messaging app, etc.
• Camera functionality is not advanced in low-end phones.
• If your app needs to send geo-coordinates, then the battery of a low-end phone drains out.
• Owing to small screens, tapping on icons scrolling, or navigating between screens can have issues, etc.

When I was testing my app on low-end devices, the following are the problems that I observed on just low-end phones and no other phones:

• Our app had a web service to send the geo-location every 10 minutes but as the low-end phone (that we were using) had 2G, the web service call was failing. Hence 2 major issues showed up – [i] the location wasn’t recording in the DB and [ii] the battery of the phone was draining out.
• While testing on 2G, we found that the web services were taking a little longer time to log in or fetch data for a screen or send data. Sometimes to see updated data it took even 15 minutes.
• We had cards to show the deliveries list for a day and while scrolling a long list of say 50 cards, the app crashed.
• While clicking the Back button, the navigation was failing and it was navigating 2 screens back rather than one screen.
• We had an image capture facility and after capturing the image we were re-sizing (reducing) the image. On the low-end phones, the camera crashed after taking 3-4 images in a go while re-sizing because the memory wasn’t getting freed from the previous image capture.

Following is an example of such an error that can come up:

• Some icons appeared too small to click and thus it blocked the testing of some functionality.

These are some of the most important failures which did not appear even once on the high-end device as per our team’s observation. All these issues were then fixed by the development team and we did a full regression on the low-end phone.

I remember that in the first release, we received a lot of camera crashing issues from customers who were using low-end phones and after clicking around 20-30 images we could reproduce the issue. It was a tough challenge for us to work on as we were at the start of our app and the mobile application was completely a new one to us.

The delivery people were not using a high-end phone and only around 5% of them were having iPhone 5 or S3 (which was the latest at that time. While testing on a low-end device it would be great to use the mobile network rather than a high-speed Wifi.

After the first release we did our scalability, stress, and UI testing on 2 low-end phones and 1 mid-end phone, however, the issues were mostly found on the low-end phones. Just to be sure, we did a BVT on a high-end phone.

You don’t need to come across the same kind of issues but you will surely observe issues with low-end phones which you will never come across on high-end phones.

Another example that comes to my mind is that a lot of apps use Google Maps to load locations or show their location.

On a low-end phone, the version of Google Maps may be old and hence there may be compatibility issues when your app makes a call to load a location in Google Maps, or due to a slow network, it may take a longer time to load. It may also be possible that the Maps may not load the exact location that your app wanted.

Emulator or Simulator as a Replacement for a Low-End Phone

Can I use an Emulator or Simulator as a replacement for a Low-End Phone?

It is not advisable to use an emulator or simulator for low-end phones because the development team of my app tried that and they were doing their BVT on the emulator hence they missed a few crashing issues.

Initially, when we started working on a mobile app, it was new to us and the development team wasn’t aware that a lot of error messages are suppressed by the emulators and the errors were not shown. We found the issues while testing on real devices.

After that, even our development team started using a real device for their testing rather than an emulator or simulator. Another reason to avoid using emulators is that it is not possible to create a 2G or 3G network and they use the high-speed network of your laptop or desktops. However, you can do UI and navigation testing on an emulator.

Testing Strategy for Low-End Phones

As a QA, what should be my ‘testing’ strategy for low-end phones?

To decide the testing strategy for low-end phones be very clear about your app requirements and what features of a low-end phone will affect your app and its performance. The defects found on low-end phones are very hard to find and tricky as well and more than that for a QA it is very difficult to reproduce, verify, and regress such bugs.

Following are some pointers to decide on your strategy, these helped and my team hopes they will help you as well:

• Never do a single hand test on a low-end phone, the more the number of minds involved the more the chances of finding hidden bugs. Hence try ‘QA pairing’ or testing in a circle like if QA1 Tests functionality F1, then to have QA2 verify the bugs reported for F1, have QA3 do the regression testing.
• Discuss your strategy, and your findings with the other QAs of your team, this helps a lot because maybe they see something that you may not or may have some experience that can help you.
• Until your app doesn’t stabilize, have a separate testing task created for low-end phones and keep them estimated.
• It is practically not possible to create an ideal test environment and also not to have several phones. Hence (with the agreement of your Manager or SM) check with the other colleagues in your company who have such devices and have them install and use your app.
• The same is true for Beta testing as well, have your BAs or Product Owner use a low-end phone and do one round of end-to-end runs.
• When the major bugs are found (like the ones mentioned above), do sit with the assigned developer while they fix and also understand from them the reason behind the bug. This will help you not only in verification but also in regression. You can also take opinions from the developers about how to regress.
• Last but not least (in my case, that was the first mobile app project) document such bugs because it is a big knowledge that you gained and can help others as well. Document how the bug surfaced, and how was it fixed, verified, and regressed.

Conclusion

The mobile application is a fast-growing market and there is a new OS phone launched every other month, hence being a QA it is not possible to test for every OS-phone combination and it is not feasible either. Therefore while choosing your testbed, make a smart choice and select the phones, emulators, etc for each category of a high, medium, and low end.

Testing on a different category may vary according to your app requirement but as per my experience, there’s no harm in testing (end to end) on a low-end device. The reason is the outdated features, slow network, etc resulting in issues that you may not observe on high-end phones.

In Mobile application testing, crashing issues are not always found testing, and sometimes customers report issues that you never noticed. Hence your Product Owner may once or twice excuse missed issues but not every other time, so being a QA try to test more on low-end phones.

If your Managers or BAs are not interested in having low-end devices as a testbed, try to convince them and be firm, unless they have strong reasons or surety that the target customers are all using the best phones.

To learn more about the importance of  Field Testing, take a look at our upcoming tutorial.