Beta testing is perhaps one of the most important steps in mobile application development.
Unlike standardized iOS, mobile devices running on Android significantly differ from each other, not just by the installed version of the operating system, but by the type of programming language, screen resolution, hardware etc.
That is the reason why you have to keep not only a team of testers in the office but also a range of devices for application testing too. Usually, this process is conducted by hand.
What You Will Learn:
At first, you need to install the application, then carefully check the usability and functionality for bugs. And this process is repeated over and over again for every device that is supposed to run the application. That’s why device management becomes so important.
Significant disadvantages, in this case, are the time-consuming work of the testers and especially the lack of devices you have running on Android.
I decided to compare 4 of the most popular services, which provide application mobile beta testing.
I tested the following third-party Mobile Application Beta Testing services
The picture below describes feature set which is the most important to me personally and its availability at different mobile beta testing platforms.
There are two main types of testing services: crash reporting services and beta testing outsourcing services. Basically, the latter includes the same functions of the former.
TestFlight is a service that simplifies the testing of applications for iOS-devices by easing the device code collections. One of the main advantages is the possibility to see how many times the app was launched, how many times it failed, and get some information for debugging.
In general, using this tool is a very pleasant experience, allowing you to efficiently collect UDID-s and distribute builds applications.
- Quick and simple files uploading via Mac OS desktop utility (actually the simplest one among all other services);
- Remote logging;
- It's free.
The main disadvantage of TestFlight:
- It doesn't support Android.
To get started with TestFlight, simply fill out their registration form. Make sure to specify that you are a developer in case you are – this will give you an access to upload your own builds and invite your own testers. The further process is pretty simple – you will need to create a team, upload your build and invite your testers.
Due to the recent TestFlight absorption by Apple, many of the developers had switched to HockeyApp in last months. Crash reports are very detailed (e.g. complete stack trace with server symbolization and although not as good as TestFlight, their desktop app provides a good way to upload the application and its dSYM file (needed for symbolication). However, the HockeyApp tools are still not free, you could only have a free trial.
- Open-source libraries;
- Build management, device management, testers' activity tracking;
- It gave me detailed crash reports.
The main disadvantage of HockeyApp:
- It doesn't support test cases. Although not every team have a need in this feature, to me personally this played a crucial role.
To get started with Hockey you will need to pay – the minimal plan for 5 apps will cost you 10 USD per month. They also have a trial plan (quick tip: to get one make sure to specify that you are a developer when registering). When the registration process is completed – register your device on Hockey, upload your app and let your testers install it.
Actually, if you aren't very keen on the beta distribution of your app, then you should definitely consider Crashlytics. This tool offers more features such as a feedback forum. This service is free right now and it seems it will stay that way in the future.
- It's primary a crash reporting tool. If you are looking for crash reporter – this one will give you all the required functionality;
- It's free.
The main disadvantage of Crashlitics
- It doesn't allow build distribution or device management, it's just a simple crash reporting tool.
To get started with Crashlitics, you will need to login with your Twitter account. Yeah, it seems that those guys are planning to gather as many social contacts as they can. However please don't expect that you will register and get an access to all the functionality instantly – you are now on their waiting list. Please wait for the further invitation. Wait. Just wait.
As well as Crashlytics, the Ubertesters platform is free. It is intended for all kinds of mobile apps. But unlike the Crashlytics it provides a wider range of services comparable to HockeyApp, including crash logging, screenshot editing, test cases support, etc.
Under one platform it combines a smart testing tool and possibility to hire a remote team of qualified testers. The real-time push notification feature lets the tester know about the new build, allowing him to install it on his device. Also, they have the ability to submit bugs and other issues while on the go from within the app itself and fully edit the screenshot of where the bug occurred. This also seems to be a good solution for PMs who wish to have a total control over the testing process.
- Screenshots editing – You can simply mark a bug area using their drawing functionality. Not a joke – this extra-small feature saved over 50% of our internal testers time;
- When short on testers you can hire their own beta testers instantly. Although I have never used this opportunity;
- Test cases support – As I mentioned above, this feature is important to me;
- It has really completed set of features that allows me to use only this platform for all my beta testing needs today.
The main disadvantage of Ubertesters:
- Too complicated user interface… It took me some time to adjust to it.
To get started with Ubertesters, go to their website and fill out a registration form. You can choose if you want to use the platform for free or want to pay (depending on your testing needs).
Quick tip: they currently offer their Pro plan (that includes all the features) for free (as a trial), so I would recommend you to choose Pro rather than Free one when choosing your plan. When registered, you will need to create your organization profile (or to join an existing one), integrate their SDK to your app, upload the first build and start testing.
So, there are few significant differences between the services listed above. Each one of them could be used for various types of mobile app beta testing solutions. To get the best result from these testing services, you have to install their SDK and implement different parts of the code to the developing applications.
Recommended read: Mobile Testing Guide
In our next article, we will discuss more on Mobile App Development Company.