A complete guide to testing mobile applications with in-depth tutorials:
Mobile technology and smart devices are the trend now and will change the future of the world as we know it. We all can vouch for it, can’t we? Now, it will be amateurish if I list what we use these mobile devices for. You all know it – Maybe better than we do.
Let’s get straight to what this tutorial is going to be about.
The Complete List of 30+ Mobile Testing Tutorials:
Mobile Testing Introduction:
Tutorial #1: Introduction to Mobile Testing
Tutorial #2: iOS App Testing
Tutorial #3: Android App Testing
Tutorial #4: Mobile Testing Challenges and Solutions
Tutorial #5: Why Mobile Testing is Tough?
Mobile Device Testing:
Tutorial #6: Test an Android Version When It Is Taken Out of Market
Tutorial #7: How to Test Mobile Apps on Low-end Devices
Tutorial #8: Field Testing for Mobile Applications
Tutorial #9: Phone Model Vs OS Version: Which Should Be Tested First?
Mobile UI Testing:
Mobile Testing Services:
Tutorial #12: Cloud-Based Mobile Application Testing
Tutorial #13: Mobile Testing Services
Tutorial #14: Mobile App Beta Testing Services
Tutorial #15: Mobile App Development Company
Tutorial #16: Cloud-Based Mobile App Testing Service Providers
Mobile App Performance and Security Testing:
Tutorial #17: Mobile Applications Performance Testing Using BlazeMeter
Tutorial #18: Mobile App Security Testing Guidelines
Mobile Testing Tools:
Mobile Automation Testing:
Tutorial #22: Appium Mobile Automation Tool tutorial
Tutorial #23: Appium Studio tutorial
Tutorial #24: Automate Android Applications Using TestComplete Tool
Tutorial #25: Robotium tutorial – Android App UI Testing Tool
Tutorial #26: Selendroid Tutorial: Mobile Automation Framework
Tutorial #27: pCloudy Tutorial: Mobile App Testing on Real Devices
Tutorial #28: Katalon Studio & Kobiton’s Cloud-Based Device Farm Tutorial
Mobile Testing Career:
Let's begin with the 1st tutorial in the series.
What You Will Learn:
- Tutorial #1: Introduction to Mobile Application Testing
- Types of Mobile Testing
- The significance of Mobile Application Testing
- Basic Difference Between Mobile and Desktop Application Testing:
- Types of Mobile App Testing:
- Mobile Application Testing Strategy
- Recommended Tool
- Test Cases for Testing a Mobile App
- Typical activities and proceedings in Testing Mobile Application
- How to Test Mobile Applications on Both Android and iOS Platforms
- Basic Difference between Android and iOS Testing
- Key Factors in Mobile Testing
- Define your own scope of Testing
- Do not Limit your Testing
- Cross-Platform Testing
- Keep an eye on the size of your Mobile App
- Testing App Upgrade Scenarios
- Device OS may not Support App
- App Permission Testing
- Compare with similar and popular Apps in the Market
- Get an Overview of Apple’s Build Rejection Criterion
- Always be on the Front Foot
- Keep your app in background for a long time (12-24 hours)
- Performance Testing of your App
- Recommended Reading
Tutorial #1: Introduction to Mobile Application Testing
Gone are the days when the telephone used to be an appliance that sat in a corner and had to ring to get our attention or a computer was a machine only a few people used – they are now an extension of our being- a window to the world and virtual servants that do as they are told.
Computers were a rage and changed how we humans thought, behaved, learned and existed.
Nowadays, Mobility solutions have taken over the market. People don’t want to switch ON their laptops/PC for everything, rather they want their handheld devices to perform everything quickly.
Hence the mobile solutions which we deliver to our clients should be tested very well. This tutorial is intended for those people who are already in mobile testing or those who have switched to it in recent times. As we already have many tutorials on definitions of mobile testing related terminologies, we will be directly dealing with the scope of this tutorial.
This tutorial will be both an introduction and your guide to Mobile Testing. So, read through!
Types of Mobile Testing
There are broadly 2 kinds of testing that take place on mobile devices:
#1. Hardware testing:
The device including the internal processors, internal hardware, screen sizes, resolution, space or memory, camera, radio, Bluetooth, WIFI etc. This is sometimes referred to as, simple “Mobile Testing”.
#2. Software or Application testing:
The applications that work on mobile devices and their functionality are tested. It is called the “Mobile Application Testing” to differentiate it from the earlier method. Even in mobile applications, there are few basic differences that are important to understanding:
a) Native apps: A native application is created for use on a platform like mobile and tablets.
b) Mobile web apps are server-side apps to access website/s on mobile using different browsers like Chrome, Firefox by connecting to a mobile network or wireless network like WIFI.
c) Hybrid apps are combinations of native app and web app. They run on devices or offline and are written using web technologies like HTML5 and CSS.
There are few basic differences that set these apart:
- Native apps have single platform affinity while mobile web apps have the cross-platform affinity.
- Native apps are written in platforms like SDKs while Mobile web apps are written with web technologies like HTML, CSS, asp.net, Java, PHP.
- For a native app, installation is required but for mobile web apps, no installation is required.
- A native app can be updated from the play store or app store while mobile web apps are centralized updates.
- Many native apps don’t require an Internet connection but for mobile web apps, it’s a must.
- Native app works faster when compared to mobile web apps.
- Native apps are installed from app stores like Google play store or app store where mobile web are websites and are only accessible through the Internet.
The rest of the article is going to be about Mobile Application Testing.
The significance of Mobile Application Testing
Testing applications on mobile devices is more challenging than testing web apps on the desktop due to
- Different range of mobile devices with different screen sizes and hardware configurations like a hard keypad, virtual keypad (touch screen) and trackball etc.
- Wide varieties of mobile devices like HTC, Samsung, Apple and Nokia.
- Different mobile operating systems like Android, Symbian, Windows, Blackberry and IOS.
- Different versions of operation system like iOS 5.x, iOS 6.x, BB5.x, BB6.x etc.
- Different mobile network operators like GSM and CDMA.
- Frequent updates – (like Android- 4.2, 4.3, 4.4, iOS-5.x, 6.x) – with each update a new testing cycle is recommended to make sure no application functionality is impacted.
As with any application, Mobile application testing is also very important, as the clientele is usually in millions for a certain product – and a product with bugs is never appreciated. It often results in monetary losses, legal issue, and irreparable brand image damage.
Basic Difference Between Mobile and Desktop Application Testing:
Few obvious aspects that set mobile app testing apart from the desktop testing
- On the desktop, the application is tested on a central processing unit. On a mobile device, the application is tested on handsets like Samsung, Nokia, Apple, and HTC.
- Mobile device screen size is smaller than a desktop.
- Mobile devices have less memory than a desktop.
- Mobiles use network connections like 2G, 3G, 4G or WIFI where desktop use broadband or dial-up connections.
- The automation tool used for desktop application testing might not work on mobile applications.
Types of Mobile App Testing:
To address all the above technical aspects, the following types of testing are performed on Mobile applications.
- Usability testing– To make sure that the mobile app is easy to use and provides a satisfactory user experience to the customers
- Compatibility testing– Testing of the application in different mobiles devices, browsers, screen sizes and OS versions according to the requirements.
- Interface testing– Testing of menu options, buttons, bookmarks, history, settings, and navigation flow of the application.
- Services testing– Testing the services of the application online and offline.
- Low-level resource testing: Testing of memory usage, auto-deletion of temporary files, local database growing issues known as low-level resource testing.
- Performance testing– Testing the performance of the application by changing the connection from 2G, 3G to WIFI, sharing the documents, battery consumption, etc.
- Operational testing– Testing of backups and recovery plan if a battery goes down, or data loss while upgrading the application from a store.
- Installation tests– Validation of the application by installing /uninstalling it on the devices.
- Security Testing– Testing an application to validate if the information system protects data or not.
Mobile Application Testing Strategy
The Test strategy should make sure that all the quality and performance guidelines are met. A few pointers in this area:
1) Selection of the devices – Analyze the market and choose the devices that are widely used. (This decision mostly relies on the clients. The client or the app builders consider the popularity factor of certain devices as well as the marketing needs for the application to decide what handsets to use for testing.)
2) Emulators – The use of these is extremely useful in the initial stages of development, as they allow quick and efficient checking of the app. The emulator is a system that runs software from one environment to another environment without changing the software itself. It duplicates the features and works on the real system.
Types of Mobile Emulators
- Device Emulator- provided by device manufacturers
- Browser Emulator- simulates mobile browser environments.
- Operating systems Emulator- Apple provides emulators for iPhones, Microsoft for Windows phones and Google Android phones
Kobiton is an affordable and highly flexible cloud-based mobile experience platform that accelerates the testing and delivery of native, web and hybrid apps on both Android and iOS using real devices. Their new scriptless test automation helps the teams with no coding expertise to generate open standard Appium scripts with ease.
List of few free and easy to use mobile device emulators
i. Mobile Phone Emulator – Used to test handsets like iPhone, Blackberry, HTC, Samsung etc.
ii. MobiReady – With this, not only can we test the web app, we can also check the code.
iii. Responsivepx – It checks the responses of the web pages, appearances, and functionality of the websites.
iv. Screenfly – It is a customizable tool and used to test websites under different categories.
3) After a satisfactory level of development is complete for the mobile app, you could move to test on the physical devices for more real-life scenarios based testing.
4) Consider cloud computing based testing: Cloud computing is basically running devices on multiple systems or networks via the Internet where applications can be tested, updated and managed. For testing purposes, it creates the web-based mobile environment on a simulator to access the mobile app.
- Backup and recovery- Cloud computing automatically takes back up of your data from remote location making recovery and restoring of data easily. And also, the storage capacity is unlimited.
- Clouds can be accessed from different devices and anywhere.
- Cloud computing is cost-efficient, easy to use, maintain and update.
- Fast and quick deployment.
- Web-based interface.
- Can run the same script on several devices in parallel.
- Less control– Since the application runs on the remote or third-party environment, the user has limited control and access to the functions.
- Internet connectivity issues– the setup is on the Internet. Network issues affect the availability and functioning
- Security and privacy Issues– Cloud computing is an Internet computing and nothing on the Internet is completing secure, so chances of data hacking are more.
- If the application contains new functionality, test it manually.
- If the application requires testing once or twice, do it manually.
- Automate the scripts for regression test cases. If regression tests are repeated, automated testing is perfect for that.
- Automate the scripts for complex scenarios which are time-consuming if executed manually.
Two kinds of automation tools are available to test mobile apps:
Object-based mobile testing tools– automation by mapping elements on the device screen into objects. This approach is independent of screen size and mainly used for Android devices.
- Eg:- Ranorex, jamo solution
Image-based mobile testing tools– create automation scripts based on screen coordinates of elements.
- Eg:- Sikuli, Egg Plant, RoutineBot
6) Network configuration is also the necessary part of mobile testing. It’s important to validate the application on different networks like 2G, 3G, 4G or WIFI.
Test Cases for Testing a Mobile App
In addition to functionality based test cases, Mobile application testing requires special test cases which should cover following scenarios.
- Battery usage– It’s important to keep a track of battery consumption while running application on the mobile devices.
- The speed of the application- the response time on different devices, with different memory parameters, with different network types etc.
- Data requirements – For installation as well as to verify if the user with the limited data plan will able to download it.
- Memory requirement– again, to download, install and run
- The functionality of the application– make sure application is not crashing due to network failure or anything else.
Download Some Sample Test Cases for Testing Mobile Applications:
Typical activities and proceedings in Testing Mobile Application
The scope of the testing depends on a number of requirements to be checked or the extent of changes made to the app. If the changes are few, a round of sanity testing will do. In case of major and/or complex changes, a full regression is recommended.
An example application testing project: ILL (International Learn Lab) is an application designed to help admin, publisher to create websites in collaboration. Using a web browser, instructors choose from a set of features to create a class that meets their requirements.
Mobile Testing process:
Step #1. Identify the types of testing: As an ILL application is applicable for browsers, so it’s mandatory to test this application on all supported browsers using different mobile devices. We need to do usability, functional and compatibility testing on different browsers with the combinations of manual and automation test cases.
Step #2. Manual and Automated testing: The methodology followed for this project is Agile with the iteration of two weeks. Every two weeks dev. team releases a new build for testing team and testing team will run their test cases in QA environment. Automation team creates scripts for the set of basic functionality and runs the scripts that help determine if the new build is stable enough to test. The Manual testing team will test the new functionality.
JIRA is used for writing of acceptance criteria; maintaining of test cases and logging /re-verification of defects. Once the iteration gets over, iteration planning meeting held where dev. The team, product owner, business analyst, and QA team discuss what went well and what needs to improve.
Step #3. Beta Testing: Once the regression testing is completed by the QA team, the build moves into UAT. User Acceptance Testing is done by the client. They re-verify all the bugs to make sure every bug was fixed and the application is working as expected on every approved browser.
Step #4. Performance test: Performance testing team tests the performance of the web app using JMeter scripts and with different the loads on the application.
Step #5. Browser testing: The web app gets tested across multiple browsers- both using different simulation tools as well as physically using real mobile devices.
Step #6. Launch plan: After every 4th week, the testing moves into staging, where a final round of end to end testing on these devices is performed to make sure the product is ready for production. And then, it goes Live!
How to Test Mobile Applications on Both Android and iOS Platforms
It is very important for the testers who test their apps on both iOS and Android Platform to know the difference between the both. iOS and Android have a lot of differences w.r.t to the look and feel, app views, encoding standards, performance, etc.
Basic Difference between Android and iOS Testing
You might have gone through all the tutorials, I have put in some major differences here, which in turn will help you as part of your testing:
#1) As we have a lot of Android devices available in the market and all of them come with different screen resolutions and sizes, hence this is one of the major difference.
For Example, Samsung S2 size is too small when compared with Nexus 6. There are high possibilities that your app layout and design get distorted on one of the devices. Probability is low in iOS as there are only countable devices available in the market and out of those many phones have similar resolutions.
For Example, before iPhone 6 and above came into existence all the older versions had the similar size only.
#2) Example to assert the above point is that in Android the developers must use 1x,2x,3x,4x and 5x images to support image resolutions for all devices whereas iOS uses just 1x,2x and 3x. However, it becomes the tester’s responsibility to ensure that the images and the other UI elements are displayed correctly on all devices.
You can refer to the below diagram to understand the concept of image resolutions:
#3) As we have the market flooded with Android devices, the code must be written in such a way in which the performance remains steady. So, it is quite probable that your app may behave slowly on lower-end devices.
#4) Another issue with Android is that software upgrades are not available for all devices at a go. Device manufacturers decide when to upgrade their devices. It becomes a very difficult task to test everything both with the new OS and the old OS.
Also, it becomes a cumbersome task for the developers to modify their code to support both the versions.
For Example, when Android 6.0 came, there was a major change as this OS started supporting app level permissions. To clarify further, the user could change permissions (location, contacts) at app level also.
Now the testing team owes the responsibility to make sure that showing permissions screen on the app launch on Android 6.0 and above and not showing permission screen on the lower versions.
#5) From the testing perspective, Pre-production build (i.e. beta version) testing is different on both the platforms. In Android, if a user is added to beta users list then he can see the updated beta build on the Play Store only if he is signed in into the play store with the same email ID which is added as a beta user.
Key Factors in Mobile Testing
I have been working in Mobile Testing for the last 2 years on both iOS and Android Platform and all the key points mentioned below in this tutorial are from my personal experience and some got derived from the issues encountered in the project.
Define your own scope of Testing
Everyone has their own style of testing. Some testers just focus on what they see from their eyes and the rest are passionate about everything that works behind the scenes of any mobile application.
If you are an iOS/Android Tester, I would suggest you to at least get yourself familiar with some common limitations/ basic functionalities of Android or iOS as it always adds value to our style of testing. I know things are difficult to understand without citing examples.
Given below are few Examples:
- We cannot change the permissions like camera, storage etc. on app level in Android devices which are below 6.0.1 version.
- For iOS below 10.0 version, call kit was not there. Just to brief you in simple words, call kit is used by a calling app and displays full-screen view when a user is getting a call from the calling apps such as WhatsApp, Skype etc. Whereas for iOS versions below 10.0 we see those calls as a notification banner.
- Many of you might have come across problems in Paytm where your app is not redirecting you to the payment page of the bank in case you want to add money to your wallet. We think the above is an issue with our bank or Paytm server but it is just that our AndroidSystemWebView is not updated. Little knowledge about programming is always helpful for you and to share with your team.
- In simple words, whenever an app is opening any web page in it, then AndroidSystemWebView should be updated.
Do not Limit your Testing
Testing should not just be limited to exploring the mobile app and logging bugs. We, as a QA should be aware of all the request that we hit our server and the response that we get out of it.
Configure Putty to view logs or verify sumo logic for logs depending on what is being used in your project. It not only helps you in knowing the End-to-End flow of the application but also makes you a better tester as you get more ideas and scenarios now.
Reason: Nothing comes into this world without any reason. Any statement should have a valid reason behind it. The reason behind analyzing the logs is that many exceptions are observed in the logs but they don’t show any impact on the UI hence we don’t notice it.
So, should we ignore it?
No, we shouldn’t. It doesn’t have any impact on the UI but it may be a futuristic concern. We could potentially see our app crashing if these kinds of exception keep creeping. As we have mentioned about App Crash in the last sentence, this leads the QA to have access to crashlytics of the project.
Crashlytics is a tool where crashes are logged along with the time and device model.
Now the question over here is that if the tester has seen app crashing then why does he need to bother about crashlytics?
The answer to this is quite interesting. There are some crashes which may not be visible on the UI but they are logged on crashlytics. It could be out of memory crash or some fatal exceptions which may impact the performance later.
Cross-Platform Interaction Testing is very important.
Citing a simple Example, say you are working on a chat application like WhatsApp which supports sending images and videos and the application is built on both iOS and Android platforms(Development may or may not be going in sync)
Ensure to test the communication of Android and the iOS, the reason being that is iOS uses “Objective C” whereas Android programming is Java-based and due to both of them being built on different platforms sometimes extra fixes need to be made at the app side to recognize strings coming from different language platforms.
Keep an eye on the size of your Mobile App
Another important advice for mobile testers – Please keep checking the size of your app after each release.
We should ensure that the size of the app doesn’t reach a point where even we as an end user won’t wish to download this app due to its large size.
Testing App Upgrade Scenarios
For mobile testers, app upgrade testing is very important. Ensure your app doesn’t crash on upgrade as the dev team may have done mismatching of a version number.
Data retention is also equally important as in whatever preferences the user has saved in the previous version should be retained when he upgrades the app.
For Example, a user might have saved his bank card details in apps like PayTm, etc.
Device OS may not Support App
Yeah, many devices may not support your app. Many of you must be knowing that vendors write their own wrappers on the top of US and it could be possible that any SQL query of your app is not compatible with the device and hence it throws an exception and it may result in not even launching the app on that phone.
Point over here is – Do try to use your app on your own devices except for the ones you use in the office. It is quite possible that you see some issues with your app.
App Permission Testing
Next on the list is Permission Testing of mobile apps. Almost every second app asks its users for access to their phone’s contact, camera, Gallery, Location etc. I have seen few testers who make a mistake by not testing the proper combinations of these permissions.
I can recall a real-time Example when we were testing a chat app which had all the features of sharing images and Audio files. Permission for Storage was set to NO.
Now, when a user would click on the Camera option it never opened till the permission for storage is set to YES. The scenario was ignored as Android Marshmallow had this functionality that if storage permission is set to NO, the camera cannot be used for that app.
The scope extends further than what we have discussed in the above paragraph. We should make sure that the app is not asking for any permissions which are not used.
Any end user familiar with the software industry may not download the app in which too many permissions are asked. If you have removed any feature from your app, then make sure to remove the permission screen for the same.
Compare with similar and popular Apps in the Market
Moral of the story – If ever you are in a doubt, then just don’t conclude it yourself. Comparing with the other similar apps on the same platform can strengthen your argument that the functionality under test will work or not.
Get an Overview of Apple’s Build Rejection Criterion
Lastly, a majority of you have might have come across situations where your builds got rejected by Apple. I know this topic won’t interest major portion of the readers but it’s always good to know the rejection policies of Apple.
As a tester, it becomes difficult for us to cater the technical aspects but still, there is some rejection criterion which the testers can take care of.
For more information on this, please click here.
Always be on the Front Foot
Being a tester, don’t let things pass over to your court from the Dev Team/ Managers. If you are passionate about testing then “Always be on Front Foot”. Try to engage yourself in activities which take place well before the code comes to your bucket to test.
Most importantly, keep looking at JIRA, QC, MTM or whichever is used in your project for all the latest updates on tickets from clients and the Business Analyst. Also, be ready to share your views if you require modifications. This applies to all the testers who are working on various domains and platforms.
Until and unless we don’t feel the product as our own, we never ought to give suggestions for new improvements or changes to the existing functionality.
Keep your app in background for a long time (12-24 hours)
I know it sounds weird but there is much logic behind the scenes which all of us don’t understand.
I am sharing this because I have seen the app crashing after launching it, say after around 14 hours from background state. The reason could be anything depending on how the developers have coded it.
Let me share a real-time Example:
In my case token expiration was the cause behind it. For one of the chat apps if launched after 12-14 hours would be stuck on the connecting banner and would never get connected until killed and relaunched. These kind of things are very difficult to catch and in a way, it makes mobile testing more challenging and creative.
Performance Testing of your App
In the mobile world, the performance of your app impacts the extent to which your application is getting recognized worldwide. As a testing team, it becomes too important to check your app response and more importantly how it works when a large number of users are using it all together.
Let’s talk about PayTm.
You all must have clicked on ADD MONEY option in the PayTm app, which then displays the balance you have in your wallet. If we consider what is going on behind the scenes, then it is a request which is going on to the server with the PayTm UserID and the server sends back the response with the balance in your account.
The above case is only when one user has hit the server. We need to make sure that even when 1000 users hit the server, they should get back the response well on time because end-user usability is our prime goal.
I would conclude this tutorial by re-iterating that mobile testing seems to be very easy in the beginning but as you keep digging in you will understand that it is not easy to ensure that whatever developed will run smoothly on thousands of devices all over the world.
You would mostly see the apps that are supported on latest and last few versions of OS only. However, it becomes the duty of the testers to ensure that they don’t miss out any scenarios. They are many other points which need to be taken into consideration but I haven’t mentioned those already iterated in the other tutorials.
Scenarios like battery consumption, interrupt testing, testing on different networks (3G, Wi-Fi), testing while switching networks, monkey testing of mobile apps, etc are all useful when it comes to mobile testing.
The attitude of testers matters a lot when it comes to the real testing environment. Until and unless you love your job you won’t bother doing things which are mentioned in the tutorial.
I have been in this field for around 6 years now and I'm very well aware that the tasks get monotonous at times but there are many other things that we can do on our own to make those monotonous tasks somewhat interesting.
Designing the right test strategy, choosing the right mobile simulators, devices, and mobile testing tools can make sure that we have 100% test coverage and help us include security, usability, performance, functionality, and compatibility based tests into our test suites.
Well, this has been our effort to fulfill multiple requests from our readers on a mobile application testing guide.
Authors: Thanks to Swapna, Hasnet, and many other mobile testing experts for helping us compile this series!
In our next article, we will discuss more on iOS App Testing.