Mobile Testing is really challenging domain. Let’s see why.
While searching for educational apps/games for my toddler, I found a game where a kid can do different saloon operations on vehicles like washing, cleaning, painting, tattooing (vehicles too can have tattoos!) and the way it was presented, I loved it.
I immediately downloaded the paid version and gave it to my toddler with a hope that I will be allowed to work for next 15-20 minutes while he will be busy decorating vehicles.
He excitedly accepted and started exploring (because kids never believe in scripted testing :) ). Within 5 minutes, he came back complaining – this is not working, please start it again…..
I saw a pop-up saying “Unfortunately the “XYZ” has stopped working”. I pressed OK and started it again and gave it to my son. Within two minutes again same complain and same warning. It was becoming interesting. Carrying a tester’s mind always, I started the investigation.
I asked my toddler what he did. (How did you find the issue?)
Nothing, I was just playing – he said.
I rephrased my question – how did you play the game, can you show me? (Can you reproduce it?)
He started playing and I watched. It was working fine and I grew impatient. I was about to turn to my work and I heard the same complain. He rejected the game saying he did not like it because it stopped for so many times.
Now the tester in me awoke and I settled myself in a chair and started playing the game. It worked fine for 10 minutes and I did not see any crash pop-up. I checked multiple tapings, data connection on/off, GPS on/off, portrait/landscape mode, power on/off but really did not find anything. Tired and frustrated, I asked my son to play it again as I wanted to see how the game stopped for multiple times for him.
One more playing round started. He selected a car from a range of vehicles to play with. He applied the little soap and then sprinkled water via designer fountain, dried the car and moved to next section where he can decorate it. He painted a car with green colour. But then he thought he did not want the green colour. Rather than changing colour, he went back to the cleaning section and sprinkled water on the car in a hope that car would gain its original colour. At this point, the pop-up blinked on the screen “Unfortunately the XYZ has stopped working”. I almost jumped out of my chair. After following same steps again, I too could see the crash.
What was happening?
After some combinations of trial and error, I found –
When the user was coming back from decorating to cleaning section, the application was crashing.
OMG, what a silly bug, I told to myself. While doing web application testing for more than eight years, I have been executing some test cases by default. One of it is – go forth and back and then again forth and do some activity. Didn’t the mobile game developer take care of this silly thing?
Ok, the developer forgot about it. Didn’t the tester find this silly mistake which made the kid to reject the game?
Why did I describe the case?
In the era where mobile is dominating all other domains and almost every fortnight a new mobile phone is launched, mobile testers are under a huge pressure. Most of the interview questions for mobile testing does not include any question about basic testing or even functional testing.
Mobile Testing, is it really tough?
The perceptions about mobile testing are – The tester should know about how to do memory leakage testing, security testing, interruption testing, device-specific testing, OS-specific testing and should be knowing one or two tools to check performance. Well enough?
No.
Agreed that mobility is really a challenging domain. Reasons of it are –
- The leaps-and-bounds growth of the mobile industry has pushed every industry to launch their app for mobile. Today, if you look for, you will be able to find apps for almost every known keyword.
- Mobile has become so much part of life that we reject or decline a site or application which is not mobile. We want everything at our fingertip. And these huge demands have created tons of stress for every project to be mobile compatible.
- A large number of smartphones in the market along with the addition of the same on monthly basis makes testing coverage a critical factor and pain point for everyone involved.
- With time, more and more OSes are being built up independently, in addition to traditional OSes like iOS, WebOS, Blackberry and Android. And most of the apps developed today are and expected to be cross-platform and therefore thorough testing on different supported OSes is required.
- Mobile testing automation is still in its childhood and even though there are numbers of tools available, mobile testing automation testers are not easily available.
- Mobile testing comes up with other different challenges like networking issues – poor, good network, network changes like Wi-Fi to 3G/4G and vice-a-versa, performance while multiple apps are being used simultaneously, memory leakage issues, battery consumption issues etc.
When mobile comes to picture, the most Important feature is user-friendliness. No matter, how good your application is, display-wise or feature wise, if the user is not able to handle it on that little screen for more than 5 seconds, he will never come back. Mobile users are spoiled with a number of options and therefore users never bear any bug.
You might think at this point – Web testing is also the same. User friendliness is required over there too. Correct. But computer users are patient than mobile users. Even as a tester, you would see that different behaviour in yourself – when you test any website on desktop, you can wait for 2-5 seconds whereas, on mobile, you keep tapping the loading symbol to kill those 2 seconds.
Conclusion
So, mobile testing is not tough because testers have to think about vast coverage and real-time usage (yes, they are surely the differentiating factors from any typical testing) but mobile testing has become tough because while running a race with allowed time to do testing, most of the mobile testers prefer to do interruption, memory leak, performance testing but avoids/gives less importance to basic functional testing (This statement is a general statement and should not be taken as a commenting weapon :) )
If you have worked in different domains, you must know that no domain is easy and every domain has its specific tough factors. Mobile domain too is not different. Yes, it’s different and it has changed the equation of requirements and users’ expectation in many ways. But it’s tough because testers think testing coverage for a number of devices is more important than thorough functional testing on at least one device.
Again, whatever I have mentioned above, are my personal thoughts and they are not written for any specific person/industry.
About the author: This post is written by STH team member Bhumika Mehta. She is a project lead, carrying 10+ years of software testing experience. She is totally into testing and loves to test everything exists.
In our next article, we will discuss more on Android Version.
Growing mobile market is daily adding more and more challenges for app testing.
Great article
This post explains how important test environment is, please do a post explaining what are the different environments that are required for mobile testing.
This is really a great article. Can you please share some of the points from your experience how to get the domain knowledge of mobile testing & which languages a typical mobile tester should learn to learn automated tools for mobile application testing?
Useful information. Thanks for sharing the article like this.
Could you post the checklist for mobile testing, where we have to concentrate on mobile apps testing (Android, iOS, Blackberry & Windows..etc).
Thanks,
Mohan
Nice that you created an article and lessons for others with a small incident.
Very useful information for the present scenario
While this issue is particularly magnified in mobile testing, it also exists fairly heavily in non-mobile domains as well as lead time to test before release is drastically reduced in an effort to be bright and fresh and new on the market, ahead of the competition, etc.
Add in the complexity of automation that is being sought, and the fact that most fast testing, be it automated or manual, still focuses on functional test, with ad hoc/exploratory or black box testing being pushed to the back, you run the increased risk of finding apps, games, and even websites that are functionally correct, but have had very little actual human interaction before being released.
This is the primary reason I added ‘Consumer Advocate’ to my resume, as I look at Golden Path, and try to approach every app as users would approach, and test as if I was trying to use the app as the intended target.
Great article… :)
In an agile/scrum world I’d say that this sentence makes more sense:
“it’s tough because PRODUCT OWNERS think testing coverage for number of devices is more important than thorough functional testing on at least one device.”
I am constantly pushing back on POs around this. Same can be said on desktop browsers – the “let’s cover all” mentality. What is your core market, who is using it, and most importantly who are you selling to.
Good article
I was fan of candy crush in 2013 when i bought new smart phone .
There were max 5 lives for playing game and after every half an hour a life will get added.
So when i lost all my lives in game, i remove synchro of my ph in setting and set the timing such that 5 lives get added in game .
i have doubt here whether it is called issue or not ?
Agreed to the points.
There are a lot of things to be considered in Mobile Testing.
Also one of the issues that I have noted in the Industry is that the assumption – less Resources are required to do Mobile/App testing.
Which again burdens the Tester, so in Less time more work and Quality work which comprises of all types of Testing methods are expected.
Thanks for writing an article and sharing the mobile testing experiences.
Very Good and True Article Bhumika… So many important points are hidden in this article.
All readers,
I am glad that the article is well accepted and informative. Thanking all of you for continuous readership and stay tuned for more like this :-).
Happy Testing.
—
Bhumika Mehta
Hello Bhumika mam,
Can u please ans my question that is mentioned in my previous comment
Tejaswini, it’s not an issue. Because when you say the next life is adding after half an hour then if you set the timer then it will see the time difference and add lives. You can test the app by setting time less than current time say 1 hour or 2 hours late and if you get lives then its an issue. Any type of app say mobile or website works on logic not on change operation you do….
When you test android, I guess the coverage is also more important as a feature working in one device might not work in others. Even though Google has tried hard in eliminating fragmentation away, the OEMs add a lot of software on top of stock android and these features might not work on all the devices. This is the reason why we should also go for device coverage as the app under development might not even work and we will use potential users of the app if they do not work.
So, its a fine line where as testers we will need to decide how important the feature is against how important device coverage is.
I agree with you Venky. But is it practically not possible to test an app in all the android devices. For a company we can imagine but individually we can test in one or two devices. In this case what is the solution to make sure you app will work fine in most frequently used devices ???
Hey Biswa, we have to constantly engage the users. Have a good feedback process. Test the app in most common devices and listen to the users. For individual app developers, I would suggest to create a community where similar developers can join and share their test devices and even test each others’ apps.
10 to 15 minutes with a device and you will be able to understand if it works in the device or not. I guess only the edge cases will be missed and this will give you a confidence as a developer / tester. :)
informative article, covering all aspects one should look into.
Hello
Well the Article is good, following the Title of article.
For me it would be appreciative and encouraging to new Tester if you will also add some help, notes, Suggestions, Idea…about how to handle this tough thing…”Mobile Testing”.
Very useful information. from such a small incidence we can learn so many things
Very nice and useful article.Can you please tel me that in which of the testing methods we can find out these type of issues.
Very true: The mobile testers emphasis on non functional testing rather than doing a basic testing on one device. I know it is also important and it would be very useful only after the basic functional testing.
mam i need deep info like pin to pin as iam btech first yr iam nt getting topics clear bt i would like to know abt software testing so plz give a notes on testing r else video
Its was really good article…
Hi, Thanks for sharing a nice post!! Mobile devices have become a crucial part of our everyday lives. With constant increase in the variety of the mobile devices, testing mobile applications has become complex and tough. However, approaching a cloud-based Mobile Testing provider might be an appropriate option. Because this type of provider can help in initiating testing activities in a short notice. But some of the key considerations to be kept in mind when choosing your cloud-based Mobile testing provider. Such as, Provider should help you out in identifying, What should you test? When should you test? What devices should you test on? When will you use automation? And relevant mobile testing specialists. Ultimately, this expertise even helps to improve ROI.
Very nice and informative article.
Good Info…I think Mobile Ecosystem and Device Fragmentation some times makes it more complicated. However this is not the case all the time. Like some times you are asked to perform testing on just few target devices.
Hello,
Thank you for submitting article. Its very helpful information.
this artical is very useful is there any way to make a flow to test mobile app testing please gove a reply for how to make smoother flow for mobile app testing