How to Test an Android Version When It Is Taken Out of Market?

The mobile application industry picked pace around 2010 and today it is a dynamic, evolving and widely spread industry.

The mobile apps have set foot in almost every field and they even play an important role in our day to day activities. Prospering in the mobile industry is a tough challenge because of the plethora of devices and the operating systems that are available in the market.

In this tutorial, I will shed light on how and what strategy should be followed when an Android version is taken out of the market.

But before that, let me give you a brief introduction to mobile app testing.

What is Mobile App Testing?

Mobile app testing means testing a mobile app for its functionality, consistency, and usability. This can be done manually or by using automation tools as well.

When a mobile app is developed, it is tested against a matrix of operating systems vs device models along with functional and non-functional testing. With the consent of the customers and the business analysts, a list of the desired operating system and the device models are prepared.

The test beds for testing comprise of the actual devices, emulators, simulators, cloud etc.

For more detailed information about mobile application testing, please refer to our tutorial  Beginners guide to mobile application testing.

Issues Arising when a Version is taken out of the Market

What happens when a version is taken out of the market and how will it affect your app?

Once a new version hits the market after a brief period of time the old version may be pulled out of the market. This happens in order to increase the business from the new versions so that the people will climb one step up and move to a higher version.

This is always done after finding out the statistics of the usage of the oldest version. When a version is taken out of the market, the share of the users using that version is very less maybe even below 15%.

On the contrary, when a new version is launched, it will not be available on all the phone models but only on the latest ones and not every person in this world goes out to buy a new latest phone to have the latest version and its latest features.

Hence ‘pulling out a version from the market’ is done after analyzing the user statistics and the support required to give for the version.

There is always a huge difference between the latest and the oldest version and hence the oldest version is pulled out when there are intermediate versions available with slightly advanced features so that people will have affordable options.

For example, as per today’s statistics, users of Jelly Beans is not even 10%, it is only a 7.6% share while the majority of the users are using Lollipop and Marshmallow while the users of the latest version ‘Nougat’ is only 13.5%.

When a version is taken out of the market the users have to necessarily upgrade to a new version of Android. They depend on the service provider or the phone company like LG, Samsung, Nokia etc., to provide a version update for their device.

The phone manufacturing companies are also smart and they do an analysis if they should give a software upgrade or not and if yes, what will be the impact on their business. There are noticeable differences between the version taken out and the version next to it.

As the support for the version is no more available it is obvious that not all apps can be upgraded to your phone thus affecting your day to day work or needs.

Hence a normal user continues to use the same device with the same OS or buys a new phone altogether which will keep getting updates at least for a few years or people root their phone and put the latest available OS for the phone model which is very risky and not recommendable.

From an application point of view, what and how does this affect your customers and business?

First and the most basic step is the analysis of the effect of the Android version that has been taken out of your application. Some of the common features of Android that have a great impact are explained below.

Features of Android that vary with Versions

Following are some of the very common features of Android that vary with versions and they are the ones that are commonly used while developing an application too.

Those include:

#1) UI

With the launch of every new version, the UI changes and is always improved when compared to its previous version. The size, color, arrangement of the application icons, etc., also change with versions. The notification bars, the widgets, in-built Google apps like Maps, Gmail etc are improved in the new version.

Similarly, the color schemes, settings also change with the new version.

If you see the following images of Jelly Beans and KitKat you can notice the difference in their UI:

#2) Camera

If we look at the first most Android version released it 2008, it had a camera facility but it had no support for the resolution or the white balance nor the quality but now we have HRD+, autofocus, dual camera functionality, much improved white balance, timer facility and a lot more.

With versions, improvements came in accessing the camera as well. And we can access the camera on the lock screen also. Now camera application loads the Google+ photos and not the gallery.

#3) Network Support

Until version 4.0+ came, the support was only till 3G. After 4.0 the support increased to 4G and now to LTE. The data speed and connectivity improved over the new Android versions thereby reducing the glitches that we faced in the past.

On Froyo or gingerbread, it may not have been possible to download movies and update the apps at the same time on the service provider’s data we can do it easily and quickly.

#4) Memory

The physical memory of a phone is often constrained and hence the RAM plays a vital role in it. The Android Runtime (ART) and the Dalvik virtual machine do the garbage collection to release the memory allocated to apps.

The Dalvik virtual machine improved over a period of time to provide more information about the running apps and the memory state. Compared to Froyo or Gingerbread, the RAM Manager of the latest versions is much advanced thereby giving the users the ability to manage the memory.

Earlier if killing the app didn’t improve, the users had to restart the phone but now from the RAM Manager, it’s much easy to free the memory thus increasing the phone speed.

#5) Performance

People assume that with every new version rolled out for Android, the functionalities beneath the surface will also improve factors like performance, stability, battery life etc. But that doesn’t happen always because at times the versions are focused on improving only the functionality, look and feel.

Performance of the apps also vary with the device model like a single processor phone will be slow when compared to a quad-core processor. Along with this the UI, the apps running in the background will also affect the overall performance.

#6) Inbuilt Google Apps:

The Google apps like Maps, Gmail, Messaging, Notes etc., come inbuilt. A lot of apps are developed to use these or connect to these for communication and the most widely used among these is the Google Maps.

A Google map app is used by a lot of apps like Zomato, Uber, OLA etc and with the launch of the new version, these apps have also improved a lot more when compared to the older versions.

Based on the above-mentioned features (there may be more to this), do the following homework for your application:

• From the above-mentioned features, which is the most commonly used and how critical is that feature for your app?
• What is the difference in the feature(s) between the version been pulled out and the next version to check if there are some major variations?
• Create a list of the functionalities of your app or the UI screens that will be affected.
• Most importantly, with the help of your BA pursue the Product Owner to do a survey of customers for the version that is being pulled out.

Data Collection for the Version been Pulled Out

When a version is pulled out it is very important that you do all the research and have the data with you to decide whether to agree with your customer/Product Owner to support that version or not.

As mentioned above, make your Product owner do an in-house survey (if possible) in order to find what percentage of his customers are in the version which is been taken out.

At times the product owners won’t be ready for such surveys but you should convince your BA who in turn will convince the product owner to do the survey because if very fewer users are using that version then it is not wise to invest efforts, money and time to support it.

Along with this, you yourself need to do a research and find out what is the percentage of users who are using the version which is been taken out, for this, you can refer the Android Developers dashboard and this has all the information that is required.

At times this may be impossible to do but still convince your product owner to collect some information so that you will have some data for you to think ahead. Based on the information collected prepare graphs or pie charts and share it with the whole team.

Read articles about what will be the general effect of the version that has been pulled out, you can refer some sites like ZDNet, Techgig etc., and share your findings with your team. Mobile application testing itself is complicated, hence under such situations when a version is been pulled out a thorough analysis and homework should be done.

You can study tutorials on mobile testing challenges and solutions and why mobile testing is tough to know more about the challenges of mobile application testing.

Such analysis may not be done by the development team or the BA, Hence use this opportunity to do the analysis because believe me, it’s a very big knowledge addition for a QA. When I was working on a mobile application project, I took the initiative to do the analysis for ‘Gingerbread’.

The application on which I worked was designed for a delivery company and our Product owner refused to do a survey as it wasn’t possible to find out the Gingerbread users among their delivery people. Hence we had to do some analysis to find out the percentage of the users of Gingerbread in the market.

Future Course for Support or No-Support

If in case you have to support the version that has been taken out of your app, there are a lot of questions that you need to take care of.

Like:

• What would be the priority of support?
• Until when should we give support to the version?
• How should we manage our resources?
• Will this affect our other sprint priorities?
• Is an in-depth testing required or only a BVT would be sufficient?

These are few critical questions that the whole team needs to decide on.

Following is a list of the solutions that worked out for my team, when we faced this situation:

#1) For the first rollout, you will need to take extra precaution and the complete app will need to be tested for every single functionality/business rule. After the bug fixing cycle, a complete regression of the app on that version is required.

#2) If your app is having a nice UI, then do a test of the UI of the three categories of phone models i.e. a small screen, a medium-sized screen, and a large-sized screen phone.

#3) Have a meeting with your SCRUM master and business analysts to decide about the priority of this version related bugs and the whole team along with the Product owner should adhere to it.

#4) It is advisable to create a separate tag for the bugs related to your specific version. The bug tracking tools have the facility to tag the bugs. Also, the bugs reported by the Product owner should be tagged for this version.

This is a very helpful step because after a period of time-based on these bugs you can decide whether to continue the support or not. Hence with a tag, it becomes very easy to get the list of bugs related to the version.

#5) Initially for a month keep a developer and a QA separately to work on the version.

#6) In your sprint planning, until the first rollout, have separate user stories and tasks for the version with the discussed-decided priorities.

#7) Ideally, give support for 3-6 months and then you can create your case whether to continue support or not based on the bugs reported ‘by the users or customers’.

Depending on the frequency of the bugs found and their criticality (i.e. crashes or functionality failures), the team can take the decision. But if no major issues were reported by the users or customers and once the app seems stable you can decide to stop the support.

To sum up, you can follow the following cycle for the initial period and then again decide about the future course:

Procedure to test the Versions taken out of the Market

How to test the versions that have been taken out of the market, what should your test bed be?

For testing your app for the version that has been taken out, I would recommend using a real device rather than an emulator or simulator or cloud. If your app is mainly focused on the UI then you can use emulator or simulator but for functionality testing, a real device is always recommendable.

For testing on real devices, use 2-3 devices which will be compatible with that particular version in which that version updates have been stopped for those devices. You can also use tablets if the users or customers are using your app on tablets. Unless you have the requirement of scalability or stress testing, I would suggest not using a cloud.

Depending on the amount of testing (to be done), chalk out the test cases and have a separate suite created and (again) tag the bugs with the version name. Perform a BVT on a 2G network and not a strong wifi connection.

Also, try to test by following the whole process as a user or customer i.e. by downloading from Google Play store or at least try to do a BVT from the point of view of an end user.

Another pointer is to use a phone which has more than half of the memory used like a 3 GB phone with only 1 GB remaining etc. These are some of the factors that can be considered for deciding your test bed.

Real Examples

5 years back when I was working on a mobile application project, we came across the same situation.

Gingerbread was to be stopped and our customer insisted to offer support to the same. It was a very difficult situation for us because our app was designed for drivers and delivery people hence it was impossible for our customer to do a survey for the Gingerbread users.

The biggest challenge that we faced was that the app was crashing on Gingerbread when the user logged in. The developer team had to write a separate piece of code specifically for Gingerbread to solve the issue.

Another challenge was the UI, we created our UI based on Icecream Sandwich just like we had cards for listing. Due to this, the UI on Gingerbread has distorted for which a separate code required a fix. But the UI was not a big priority hence things that could be fixed ‘easily and quickly’ were only taken up.

The third major issue was about the location coordinates. The web service was failing to send the coordinates for Gingerbread. As this was an important feature of our app, we again added a separate code for this.

Due to such fixes, the app was regressed on the other versions as well in order to check if anything broke but fortunately it was running smoothly. We had a separate tag created in JIRA for tracking the Gingerbread issues and a QA was assigned for 3 months only for Gingerbread.

After 3 months, the app was quite stable on Gingerbread and the Product Owner agreed to keep the Gingerbread issues on a less priority.

Conclusion

Mobile application testing itself is a challenging experience and it offers a lot in terms of learning, experience, and knowledge. This is a dynamic and evolving market, that changes every 6 months. When it comes to taking out versions, it needs to be done systematically and thoughtfully, after doing a lot of study and homework.

The team should be on the same page about the decision made. The decision of both supporting or not-supporting members should be taken after accounting for your efforts, time and resources required for the same.

Being a QA, be alert and stay involved because it is you who will be testing. Study as much as possible about the versions, their features, and the handset models.

Not everyone gets a chance to work on Mobile apps so if given a chance grab it and use it with utmost concentration.

Check our upcoming tutorial to know about the Importance of testing low-end devices.