Latest Released Phone Model Vs OS Version: Which Should Be Tested First?

Sometimes QA’s face a difficult situation when the time is constrained and the testing needs to be prioritized with respect to the testbeds. In case of a mobile application, testing is unquestionably difficult to decide what to include and what not to.

When a new Nexus or iPhone or a Note is rolled out, it undoubtedly comes with advanced features, sporty UI to attract the users to buy the phone. It’s like a race if one company gives a bunch of advanced features, then the other tries to make something more advanced at a reasonable price to capture the market.

Apps which have an appealing UI will definitely need to be tested on such devices to verify that if the UI is looking lively. It is not possible to test (not even needed) on every phone model but phones like Nexus, iPhone and Note should definitely be tested.

In this tutorial, I am going to describe a very specific game plan when there’s a time crunch and as an icing on the cake, a new phone model and a new OS that is released in the market.

Latest Released Phone Model Vs Version

I have come up with this tutorial because me and my team faced this difficult situation in 2014 when Nexus 6, iPhone 6 and Lollipop were all rolled out on quite close dates. It was a difficult situation for us because we had time bound sprints and we couldn’t ask for a months’ time for testing.

Testing comparison VS newsletter

[image source]

Phone or OS: What Should be my Testing Priority?

This is a tough call that no one can make for you but only you and your team have to discuss and decide which one should take precedence over the other unless your product owner lays the priority for you.

You can ask the product owner for inputs but necessarily will they have an answer because it is new to them as well. I cannot give a direct answer to this question but I can give you few signs or pointers to take your decision.

The pointers include:

  • The first and foremost thing to do is to spend time (not more than 2 days) to find out what’s new in the OS and the phone models. Split up the research work with the other QAs and that will save your time.
  • Collect all the findings and share the same with the development team.
  • Have a discussion with the development team, BA to figure out what will have more impact on your app.
  • Have a very clear idea about the purpose of testing before choosing between the latest phone and OS.
  • If your app is a commercial app like a shopping app, then UI testing becomes important. In this case, you should check if the phone UI or the OS UI will be bumpier.
  • If your app is more of a personalized professional app like an Email app, then UI, look and feel don’t matter that much, choose latest OS testing in such case.
  • If your app is using 3rd party apps like Maps, phone dialer, phone messaging app etc then prefer testing on latest OS because apps don’t change with phones.
  • But on the contrary, if the introduction of new features such as LTE is affecting the performance of your app then you should definitely test on the latest phone. Like Jio mobile network technology needs a LTE feature to make calls.
  • It may also happen that the latest phone is released with the latest OS, which may solve your problem to some extent. But ultimately the latest OS support is a decision of the customer. They may not want to immediately support the latest OS.

To talk about my experience, we choose the latest Nexus and iPhone over the Lollipop OS because our customer was in no hurry to launch support for the latest OS.

The Nexus came with KitKat and we tested on that because our app was focused on successful logging of events related to the package deliveries. The features of Lollipop were not going to affect a lot on our app but we did want to see the performance on the latest phone.


How to Test and What Strategy to Follow?

A well-planned strategy needs to be in place when you have to prioritize.

Following are the list of items that you should be ready with before starting to test:

#1) Functionality Matrices:

Create a document with the matrices of the phone or OS features that will affect your app against the functionalities that will get affected.

#2) UI, Theme & Appearance:

It’s a 100% surety that the UI, the themes, and the appearances will be different.

Hence create a set of test cases for testing of the UI specifically. Like I have a Redmi Note 4 which has a setting that the theme changes every day, if today the color scheme is light green, tomorrow it’s blue or red etc and so with that the color schemes of my apps and their icons also varies (which I don’t like always).

Create a similar matrices document for the UI, theme etc that will have an impact on the UI of your app as well.

#3) Testing artifacts:

Be it OS testing or device testing, whichever you choose, have the test cases created against the above-mentioned matrices. Create a separate suite for this testing as it will be specifically set for the phone or the OS.

#4) Tagging:

Tag the test cases, defects and if possible the user stories for this. This will help you to filter out from a huge set of test cases or user stories for reference and sharing with the team.

#5) Using Simulator or Emulator:

For OS testing, you can test the UI on simulator or emulator although I personally don’t prefer testing on those but UI, theme etc., can be tested to save time. Avoid using emulator or simulator for testing against the latest phone.

#6) Automation:

In such situations, automation will do a great help as it saves a lot of your time that you spend on running the functional tests. There are a lot of amazing tools that will ease your day and you can concentrate on other important stuff.

To read more about automation tools, please visit our page: 5 best automation tools for testing Android applications.

Following are some indicators from my experience on how to test:

  • Create the Test Suite: Filter and add new test cases as per the requirements and keep your test suite ready.
  • BVT: Try to have automated test cases to save time and this, in turn, will help in getting quick results before accepting the build for testing.
  • Phone or OS Vs app testing: Functionality, UI, features of phone or OS testing for phone or OS. You may not need to verify each and every test case but only verify those which have an impact on new features.
  • Bug Verification/Regression: Focus more on the phone or OS feature(s) while regression and cover all the minute details related to the same.
  • Field Test: For sure do at least one BVT to verify the app for testing the end to end functionality.

You can read in details about field testing from here.

Phone OR OS

Conclusion

If there’s no priority set by the product owner then it’s an internal team decision which should be taken very cautiously. When communicating your decision to the product owner, make sure that you are listing the reasons behind your choices very clearly.

Under such situations more than testing, an exhaustive and utter knowledge about the impact is extremely important as that will define your future course and affect your decision as well. Hence do all the homework before choosing between the phone and OS. It is best recommended to create a sheet of features affecting Vs your app functionality.

Try to use automation more because it will save a lot of time and help you to spend time on other activities as well.

In our upcoming tutorial, we will learn about Mobile App UI Testing.