Mobile Testing is a really challenging domain. We will find out why mobile testing is tough in this article. Let’s get started.
While searching for educational apps/games for my toddler, I found a game where a kid is able to do different saloon operations on vehicles like washing, cleaning, painting, tattooing (vehicles too can have tattoos!) and I loved the way it was presented.
I immediately downloaded the paid version for my toddler with a hope that I can work for next 15-20 minutes while he is busy decorating vehicles.
Table of Contents:
Why Mobile Testing is Tough
He excitedly accepted and started exploring (because the kids never believe in scripted testing 🙂 Within 5 minutes, he had a complaint that it was not working and wanted to start it again.
I saw a pop-up saying “Unfortunately the “XYZ” has stopped working”. I pressed OK and started it again and gave it to my son. Within two minutes there was the same complaint as well as the same warning again. It was quite interesting. As a tester, I started the investigation.
I asked my toddler what he did. How did you find the issue?
“I was just playing,” he said.
I rephrased my question, how did you play the game? Can you show me? (Can you reproduce it?)
He started playing and I was watching him. It was working fine and I grew impatient. I was about to work and I heard the same complaint. He rejected the game saying he did not like it because it stopped so many times.
I settled in a chair and started playing the game. It worked fine for 10 minutes and I did not see any crashing pop-up. I checked multiple tapings, data connection on/off, GPS on/off, portrait/landscape mode, power on/off but did not find anything. Tired and frustrated, I asked my son to play again as I wanted to see why the game had stopped multiple times for him.
He then selected a car from a range of vehicles to play with. He applied the little soap and then sprinkled water via a designer fountain, dried the car and moved to the next section where he could decorate it. He painted the car green. But then he decided he did not like the green colour. Rather than changing colour, he went back to the cleaning section and sprinkled water on the car in the hope that car would gain its original colour. At this point, the pop-up blinked on the screen “Unfortunately the XYZ has stopped working”. After following the same steps again, I too could see the crash.
What was happening?
After some combinations of trial and error, I found –
When the user was coming back from decorating to cleaning section, the application was crashing.
OMG, what a silly bug, I told myself. While doing web application testing for more than eight years, I have been executing some test cases by default. One of them is going forth and back and then again forth and doing some activity. Didn’t the mobile game developer take care of this silly thing?
Ok, the developer forgot about it. Didn’t the tester find this silly mistake which made the kid reject the game?
Why did I describe the case?
In an era where mobile phones are dominating all other domains and almost every fortnight a new mobile phone is launched, mobile testers are under huge pressure. Most of the interview questions for mobile testing do not include any questions about basic testing or even functional testing.
Is Mobile Testing really tough
The perceptions about mobile testing are that the tester should know about how to do memory leakage testing, security testing, interruption testing, device-specific testing, OS-specific testing and should know one or two tools to check performance. Well enough?
No.
Agreed that mobility is really a challenging domain. The reasons for it are as follows –
- The leap-and-bound growth of the mobile industry has pushed every industry to launch their app for mobile. Today, if you search, you will be able to find apps for almost every known keyword.
- Mobile phones have become so much part of our life that we reject or decline a site or application which is not mobile. We want everything at our fingertips. This huge demand has created tons of stress for every project to be mobile compatible.
- The large number of smartphones in the market along with the addition of the same on a monthly basis makes testing coverage a critical factor for everyone involved.
- Over time, more and more OSes are being built up independently, in addition to traditional OSes like iOS, WebOS, Blackberry and Android. Most of the apps developed today are expected to be cross-platform and therefore thorough testing on different supported OSes is required.
- Mobile testing automation is still in its childhood and even though there are a number of tools available, mobile testing automation testers are not easily available.
- Mobile testing comes up with other different challenges like networking issues – poor, good network, network changes like Wi-Fi to 3G/4G and vice-a-versa, performance while multiple apps are being used simultaneously, memory leakage issues, battery consumption issues etc.
When it comes to mobile pictures, the most important feature is user-friendliness. No matter how good your application is, display-wise or feature wise, if the user is not able to handle it on that little screen for more than 5 seconds, he will never come back. Mobile users are spoiled with a number of options and therefore users never bear any bug.
You might think at this point, web testing is also the same. User friendliness is required over there as well. But computer users are more patient than mobile users. Even as a tester, you would see that different behaviour in yourself, when you test any website on desktop, you can wait for 2-5 seconds whereas, on mobile, you keep tapping the loading symbol to kill those 2 seconds.
Conclusion
So, mobile testing is not tough because testers have to think about vast coverage and real-time usage (yes, they are surely the differentiating factors from any typical testing) but mobile testing has become tough because while running a race with allowed time to do testing, most of the mobile testers prefer to do interruption, memory leak, performance testing but avoids/gives less importance to basic functional testing (This statement is a general statement and should not be taken as a commenting weapon 🙂
If you have worked in different domains, you must know that no domain is easy and every domain has its specific tough factors. The mobile domain is no different. It has changed the equation of requirements and users’ expectations in a number of ways. But it’s tough because testers think testing coverage for a number of devices is more important than thorough functional testing on at least one device.
Again, whatever I have mentioned above, are my personal thoughts and they are not written for any specific person/industry.
About the author: This post was written by STH team member Bhumika Mehta. She is a project lead, carrying 10+ years of software testing experience. She is totally into testing and loves to test everything that exists.
In our next article, we will discuss much more about the Android Version. What is your opinion on mobile testing? Post your views and feedback about this article in the comments section below. We would love to hear from you.
Hello
Well the Article is good, following the Title of article.
For me it would be appreciative and encouraging to new Tester if you will also add some help, notes, Suggestions, Idea…about how to handle this tough thing…”Mobile Testing”.
Nice that you created an article and lessons for others with a small incident.
This is really a great article. Can you please share some of the points from your experience how to get the domain knowledge of mobile testing & which languages a typical mobile tester should learn to learn automated tools for mobile application testing?
Very true: The mobile testers emphasis on non functional testing rather than doing a basic testing on one device. I know it is also important and it would be very useful only after the basic functional testing.
Very useful information. from such a small incidence we can learn so many things
Very useful information for the present scenario
In an agile/scrum world I’d say that this sentence makes more sense:
“it’s tough because PRODUCT OWNERS think testing coverage for number of devices is more important than thorough functional testing on at least one device.”
I am constantly pushing back on POs around this. Same can be said on desktop browsers – the “let’s cover all” mentality. What is your core market, who is using it, and most importantly who are you selling to.
this artical is very useful is there any way to make a flow to test mobile app testing please gove a reply for how to make smoother flow for mobile app testing
Good article
I was fan of candy crush in 2013 when i bought new smart phone .
There were max 5 lives for playing game and after every half an hour a life will get added.
So when i lost all my lives in game, i remove synchro of my ph in setting and set the timing such that 5 lives get added in game .
i have doubt here whether it is called issue or not ?
Useful information. Thanks for sharing the article like this.
Could you post the checklist for mobile testing, where we have to concentrate on mobile apps testing (Android, iOS, Blackberry & Windows..etc).
Thanks,
Mohan
When you test android, I guess the coverage is also more important as a feature working in one device might not work in others. Even though Google has tried hard in eliminating fragmentation away, the OEMs add a lot of software on top of stock android and these features might not work on all the devices. This is the reason why we should also go for device coverage as the app under development might not even work and we will use potential users of the app if they do not work.
So, its a fine line where as testers we will need to decide how important the feature is against how important device coverage is.
Hello,
Thank you for submitting article. Its very helpful information.
Very nice and informative article.
mam i need deep info like pin to pin as iam btech first yr iam nt getting topics clear bt i would like to know abt software testing so plz give a notes on testing r else video
I agree with you Venky. But is it practically not possible to test an app in all the android devices. For a company we can imagine but individually we can test in one or two devices. In this case what is the solution to make sure you app will work fine in most frequently used devices ???
Hi, Thanks for sharing a nice post!! Mobile devices have become a crucial part of our everyday lives. With constant increase in the variety of the mobile devices, testing mobile applications has become complex and tough. However, approaching a cloud-based Mobile Testing provider might be an appropriate option. Because this type of provider can help in initiating testing activities in a short notice. But some of the key considerations to be kept in mind when choosing your cloud-based Mobile testing provider. Such as, Provider should help you out in identifying, What should you test? When should you test? What devices should you test on? When will you use automation? And relevant mobile testing specialists. Ultimately, this expertise even helps to improve ROI.
Good Info…I think Mobile Ecosystem and Device Fragmentation some times makes it more complicated. However this is not the case all the time. Like some times you are asked to perform testing on just few target devices.
This post explains how important test environment is, please do a post explaining what are the different environments that are required for mobile testing.
Its was really good article…
Very nice and useful article.Can you please tel me that in which of the testing methods we can find out these type of issues.
Agreed to the points.
There are a lot of things to be considered in Mobile Testing.
Also one of the issues that I have noted in the Industry is that the assumption – less Resources are required to do Mobile/App testing.
Which again burdens the Tester, so in Less time more work and Quality work which comprises of all types of Testing methods are expected.
Very Good and True Article Bhumika… So many important points are hidden in this article.
Growing mobile market is daily adding more and more challenges for app testing.
All readers,
I am glad that the article is well accepted and informative. Thanking all of you for continuous readership and stay tuned for more like this :-).
Happy Testing.
—
Bhumika Mehta
Hey Biswa, we have to constantly engage the users. Have a good feedback process. Test the app in most common devices and listen to the users. For individual app developers, I would suggest to create a community where similar developers can join and share their test devices and even test each others’ apps.
10 to 15 minutes with a device and you will be able to understand if it works in the device or not. I guess only the edge cases will be missed and this will give you a confidence as a developer / tester. 🙂
Hello Bhumika mam,
Can u please ans my question that is mentioned in my previous comment
Thanks for writing an article and sharing the mobile testing experiences.
While this issue is particularly magnified in mobile testing, it also exists fairly heavily in non-mobile domains as well as lead time to test before release is drastically reduced in an effort to be bright and fresh and new on the market, ahead of the competition, etc.
Add in the complexity of automation that is being sought, and the fact that most fast testing, be it automated or manual, still focuses on functional test, with ad hoc/exploratory or black box testing being pushed to the back, you run the increased risk of finding apps, games, and even websites that are functionally correct, but have had very little actual human interaction before being released.
This is the primary reason I added ‘Consumer Advocate’ to my resume, as I look at Golden Path, and try to approach every app as users would approach, and test as if I was trying to use the app as the intended target.
Great article… 🙂
Great article
informative article, covering all aspects one should look into.
Tejaswini, it’s not an issue. Because when you say the next life is adding after half an hour then if you set the timer then it will see the time difference and add lives. You can test the app by setting time less than current time say 1 hour or 2 hours late and if you get lives then its an issue. Any type of app say mobile or website works on logic not on change operation you do….