5 Mobile Testing Challenges and Solutions

The mobile boom is obvious and it is pretty clear they are here to stay. The smart-phones are rapidly becoming the primary method of interaction for consumers and businesses worldwide, with thousands of app being generated each day. Mobile goes beyond smart-phones and tablets. Apps are now being incorporated into cars, wearable tech and home appliances

The phenomenal growth of mobile devices has opened up avenues for organizations to integrate them into the computing environment. Today’s mobile applications deliver complex functionality on platforms that have limited resources for computing. Diversity presents unique challenges that require unique testing strategies.

What will you learn in this article?

In continuation with the detailed and quite insightful article Beginner’s guide to Mobile Testing, we will here understand the various challenges which are faced when performing mobile testing and what are the current solutions available in the market to tackle them all.

According to the Cap Gemini Quality Report (Mobile Testing), 18% of responding companies say that they do not have enough time to test mobile apps, and 65% do not have the right tools. Also, 52% cite lack of devices as a reason not to do mobile testing. These all are real-time challenges faced by the organizations to successfully perform mobile testing.

mobile application challenges 1

Fig 1 – Challenges to mobile testing

Let us in here, list and understand one by one the real-time challenges for the mobile testing:

#1 Multitude Mobile Devices

Over 500m Android devices shipped since Android 1.0, about 220m iOS devices have been shipped since 2007. These huge number of mobile devices availability ranging from handsets to smartphones, to tabs, to pads and wearable tech provides a huge diversity of environments which your mobile app faces.

Further, the quality team cannot guarantee that if a tested application works well on a given device, it will work 100% on another device even if it’s from the same product family because the screen resolution, CPU, Memory, OS optimization and hardware could be different.

mobile application challenges 2

Fig 2 – Number of Devices for testing mobile apps

The question to ask in here is, “Whether testing on 5-8 devices is enough? The answer is “Yes”if reaching only 25% of the customer base is “OK” for your CEO.

#2. Device Fragmentation & Various OS Platforms

Perhaps the most difficult aspect of the mobile testing matrix is device fragmentation. Though the iOS device matrix is growing more than ever, fragmentation is especially an issue for the Android operating system. Take, for example, the image below-

(Click on images to enlarge)

mobile application challenges 3

Fig 3 – Devices and OS platforms

This is a data chart by OpenSignal of nearly 12,000 separate Android device models encountered in July 2013. (In comparison, the same chart from 2012 found nearly 4,000 devices.) If you’re concerned with quality on cross-platform apps, you’ll encounter a similarly complicated matrix. Although less daunting than the hardware matrix, the variety of mobile operating systems also poses a challenge for engineering teams whose goal is to provide a consistent user experience across platforms.

Mobile application testing is also challenging, due to compatibility issues as a mobile application can be deployed across devices which have different-

  1. Operating systems like iOS, Android, Blackberry, Windows etc.
  2. Versions of an operating system such as iOS 4.X, iOS 5.X, BB 4.X, 5.X and 6.X.

#3. Different Mobile App Types

A mobile app can be a native app, a web app or a hybrid app which has both contents. Testing of each such app type is different than another as their implementation is quite different from one another.

mobile application challenges 4

Fig 4 – Types of mobile apps

As we see each app behaviour from installation to functionality is different from one another, we understand that their testing and test coverage will also be different. For more details on it, you can refer to the Beginner’s guide to Mobile Testing


#4. Numerous Test Interfaces

Mobile emulators and simulators are the important testing tool and they enable us to verify general functionality and perform regular regression testing. The very character of emulators and simulators means testing is being conducted in an environment which is not real.

The advantages of such tools are limited in scope, and should never be considered a substitute for real-world. Using emulators and simulators in tandem with in-the-wild testing will give you the best results.

mobile application challenges 5

Fig 5

#5. Variety of Testing Tools

Summarizing the daunting complexity of mobile test automation, even more, daunting is the huge availability of mobile test automation tools in the market. Free/Paid. For native app or web app? For Android or for iOS, so which is the one automation tool for your mobile test automation needs, or the question is – Is there a single tool for our Mobile Test automation needs?

mobile application challenges 6

Fig 6 – Mobile test automation tools

Solution:

So what is that ideal solution for your mobile testing needs? What are the different measures you ought to take up? Let us have a look at some of them –

#1. Mobile Test Lab

If your mobile test needs are immense and often, a good idea is to invest in creating your own mobile test lab.

mobile application challenges 7

Or one can as well look for solutions to hire an external test lab or explore the cloud mobile lab solutions. Many companies have already started venturing into it. Some of the websites of these companies are –

Mobile App Testing Labs in Secure Private Cloud:

One ought to create a proper sampling of effective mobile device availability which appropriately covers the larger segment of the variety needed. You are required to have this lab to ensure that you have not simply tested your app on an emulator or simulator but on the actual real device. Also, you have catered well the variability of OS, device type, fragmentation, screen, memory and other factors which can impact your application functioning on the mobile device.

#2. The Right Testing Solution

So what exactly are you looking for in the mobile testing solution? An IDE so that scripting is reduced, a keyword-driven approach so that even manual testers can effectively use it, integration with a mobile cloud to help you overcome device challenges, a tool which helps you identify object both based on property and on an image as well. Many solutions are available in the market- Robotium, Appium, Calabash from open source, and in commercial – eggplant, perfecto etc.

Below given is a comparison table, taken from an informative blog, to help you understand the features available with tools-

(Click on images to enlarge)

mobile application challenges 8

Table 1 – Mobile Test automation solutions comparison

Conclusion:

The above article is an extension to the beginners, where the aim was to specifically highlight the real-time challenges faced in the world of mobile testing, and present a comparison matrix of these solutions to help the tester make an informed choice about the approach and solution for their mobile test needs.

About the author: This is a guest article by Pallavi Sharma. With overall 9 years of professional experience, she is in the software testing field for the last 7.5 years in the domain of functional, non-functional testing and automation tools like Webdriver, QTP, Sahi, Watir, SoapUI, openSTA, and RPT. Now she is handling lead and project management roles of varied teams following the SCRUM methodology.

What other is mobile testing challenges you facing? And did you find any solution for that? Please let us know in comments.




Recommended reading

25 comments ↓

#1 Mohammad Faizan

This Article was almost perfect but i would like to know one tool that will be good for mobile app testing and will also helpful with career perspective..

#2 Rohit

testing on minimum 399 devices for 90% coverage manually is scary.

#3 Supriya

yes can you suggest one good tool for mobile test automation? or may be one per category if there are many..

#4 mobile app tester

you probably forgot the most meaningful test clouds out there. Testdroid, Apphwack, DeviceAnywhere.

#5 Pallavi

@Mohammad Faizan, well appium is a good tool for open source, in combination with Sikuli. For commercial tools, many options are available to learn. You can pick SeeTest. A lot also depends on your project needs etc.

Hope it helps. And thanks a lot for the appreciation. Glad you liked it.

#6 Pallavi

Hi @Mohammad Faizan, thanks for the appreciation, glad you liked the article. Well in the open source the one tool you should know is Appium, and for commercial, you can learn SeeTest, as it covers both ios and android platforms as well different app kinds.
A lot also depends on your project needs. Let me know if you wish to discuss more on it.

#7 Pallavi

@Rohit, yup it is definitely, and thats what the stats says. All the more reason to go for automation, if one can!

#8 Pallavi

@Supriya, I would suggest to have a look at the table1, as it exactly shows it. In open source Appium is great tool to know. Hope it helps.

#9 Pallavi

@mobile app tester, thanks for the feedback. Point taken, you are right they are indeed good cloud solutions.

#10 Ameet

Any good institutes in mumbai and bangalore which provides Mobile testing course ?

#11 Saurabh

Hello Sir,

Which tool is used for location sync from mobile for load testing.

#12 Jonathan

Great article! I would also add Applitools Eyes (http://applitools.com). We use it with Appium to test our mobile apps on multiple devices and it makes it really easy to build automated tests with significant coverage.

#13 manu

Any one can help me to known the interview questions on Multifunction Printers

#14 Rahul Thakur

What type of testing strategy should be used for mobile applications for cross platform….

#15 Lee

What about mobile app automation frameworks?
who has automation frameworks for android and ios?
thanks Lee

#16 Hannes Lenke

Really Great article. I’m wondering if you can add https://testobject.com to your mobile device cloud list?
Disclaimer: I’m co-founder if TestObject.

Cheers,

Hannes

#17 vineet

hi Pallavi,

I have an exp of 3 years, and now I am planning to shift to automation, selenium. I have completed a selenium course as well but I need a link where from I I can understand it better . kindly help.

cheers

#18 Vaibhav Agarwal

Hi Pallavi,
How do we know on what all OS versions we need to test.
Also for what all screen resolutions we need to consider?

#19 Lavanya

Hi Vaibhav,

This is one of the first challenge which comes in our mind before starting Mobile app testing

There are different approached different people follow

We follow two ways to decide it:-

1.Based on market research(Decide your region of app launch )-You can test on most popular devices of that region.

2.Decide various mobile aspects(Screen size,Network,Memory,manufacturer etc) and categorize your devices on based of these aspects.

#20 Dirk de Kok

Good mobile testing is such a pain with tools being still relatively immature. I see small agile teams here in Silicon Valley with tight deadlines that don’t want to spend a lot of time having to learn new imperfect testing tools. Gaming is impossible to automate. Functional quality is also not anymore seen as the only quality, as product market fit is even more important. This leads companies to want to push their products out as fast as possible and learn from their users’ feedback. This means that there is a lot of opportunity for manual testing, and on the go. Look for smart testing companies that don’t require elaborate testing plans and can send you results in 24-48 hours.

#21 Santhosh Tuppad

Interesting! I also find this mindmap http://apps.testinsane.com/mindmaps/challenges-with-mobile-application-testing interesting :)

Just my few cents! :-)

#22 Ankur Surothia

Hi All,

I have assigned task in which i have to take out all the possible challenges we face while doing the IOS automation can anyone please please suggest.
And i quit new to this field.

#23 Alisha Henderson

Very nice article.

Thanx for sharing this article.

The challenges that you have mentioned is true and yes mobile testers face these challenges on daily basis. I am also software tester and I also face these.

#24 Mehdi Hussen

Have seen this article helpful for me as a budding Software Engineer. Daily new Learning experiences. Would like to know few tools which simultaneously test app for the platform. this will really help in optimizing the app as well as improving work efficiency for us. Thanks for sharing this Beautifull article.

#25 Mukta

Hi Pallavi,

I was looking for some good website for mobile testing. I am preparing for interviews based on mobile testing. Your articles are way too helpful and easy to understand. Just like i was looking for. Would like to say thank you for creating this amazing website for testing professionals like me who want to grow in career.

Thanks a lot everyone who contributes to this site!

Most Important, i really liked the way you guys respond to our queries. That’s wonderful. Please keep responding to us. You are doing a great job out there.

I am going to be a regular visitor of your site now. So, please be responsive to my queries. : ) I am sure you would be.

Cheers
Mukta

Leave a Comment