Mobile Performance Testing is nowadays very popular among the websites who try to make their presence on mobile and portable devices.
Desktop Performance testing is something we are doing for a few years and good to understand how Mobile Performance Testing differs Desktop Performance testing.
This tutorial will help you to answer this and it too brings your attention to BlazeMeter – an emerging tool in Mobile Performance Testing.
=> Click Here For Complete Performance Testing Tutorials Series
The tutorial will touch the key features of BlazeMeter and important information to know about it before starting to using it.
By the end of the tutorial, you will able to know how far BlazeMeter meets your expectation of doing Mobile Performance Testing.
Table of Contents:
Performance Testing
Performance Testing is:
- A type of non-functional testing
- Applicable for applications having concurrent users
- Testing performed to determine how a system performs under a particular workload (users)
- To demonstrate that the system meets the performance criteria
- Measure what part of the system or workload causes the system to perform badly
After knowing about performance testing, the next thing that comes to our mind is
Why do we need performance test?
I will like to put an example of one well-known website which is improved a lot in last few months
i.e. Indian Railway Catering and Tourism Corporation used to book Indian railway ticket.
So Performance testing is required to
- Verify Application’s readiness to go live
- Verify if the desired performance criteria are met
- Compare performance characteristics/configurations of the application with the standard
- Identify Performance Bottlenecks
- Facilitate Performance Tuning
So far we were doing performance testing for desktop applications but now there is lots of focus on mobile performance testing? Why is this so?
Why is this so?
Let’s see. Some of the below facts indicate a use of mobile is increasing day by day and it is becoming part of our life now
- 28 percent of Internet usage comes from mobile phones. It is projected to take over desktop usage by 2014
- 62 percent of companies that designed a website specifically for mobile had increased sales
- About 91 percent of US citizens have their mobile device with them 24/7
- The number of unique screen resolutions has risen from 97 in 2010 to about 232 in 2013
- Only 55 percent of companies are currently conducting an online user experience testing
- 48 percent of users say that if they arrive on a business site that isn’t working well on mobile, they take it as an indication of the business simply not caring
- 90 percent of people use multiple screens to access the web
(These statistics are from stickyminds.com)
Mobile Performance Testing Vs Desktop Performance Testing
Let see how mobile performance testing is different from desktop performance testing
Platforms and Device
Mobiles have a variety of platform running on different devices. It is ensured that all major platforms and all type of devices supplied by major brands are covered. In the desktop, specific platforms are targeted.
Screen Size
Mobiles have lots of variation in screen size which is not the case for desktops.
Usability
In a desktop, user interaction is pretty much limited to the mouse and keyboard whereas in mobiles user interaction modes are a touch screen, voice command, motion detection and these are more complex and difficult to simulate.
Types of Application
Desktop applications are mostly Windows (desktop) or web-based applications whereas Mobile applications may be Native, Web or Hybrid.
Simulators & Emulators
For mobile testing, use of Simulators & Emulators is prevalent but they are still not a true representation of the devices.
Privacy & Security concerns
Most of the mobile applications use personal data such as phone contacts & messages, videos, photos and it is assumed that applications are secure and they are not accessing the data intentionally. It is also assumed that they will not share those details further.
Networks
Desktop uses LAN and wireless which is very predictable as compared to a mobile network. A mobile network could be 3G, 4G, weak signals, no signals, powerful signals and it’s difficult to simulate or switch from one to another which is very much practical.
SW Upgrade
In mobiles, revision of OS/ Platform is very common and this affects the performance of a mobile app.
Session Management and Interruptions
Apps and users are constantly interrupted by calls, SMS, push notifications, etc. In mobile testing, we too have to take them into the account.
Battery
The battery which is not-at-all in the picture for desktop plays an important role in mobile performance testing. It’s good to see the application performance on varying battery state.
Component of Mobile Performance Testing
These are major components of Mobile testing are
1) Emulators and Simulators- Mobile emulators are software platforms, installed on a laptop or desktop, that imitates a mobile platform or OS environment from the point of view of the application running on the OS. In other words, apps think they are running on a real phone, not a PC.
These are not the true representation of mobile although it helps a lot in cost-cutting.
2) Mobile Devices – Real-time testing requires real devices and there are few scenarios which are not possible on emulator e.g. how the apps behave when it is interrupted by a call or SMS.
3) Tool Selection – Selection of Right tool is the key of Mobile Testing.
4) Network – Real user uses a network which keeps changing according to the signal. It is important to simulate different networks with different bandwidth limits.
Mobile Performance Testing Tool – BlazeMeter
Here are the key features of BlazeMeter:
- BlazeMeter is a Mobile Performance Testing Tool
- The tool is highly scalable to one million virtual users
- It has comprehensive web-based reporting which could be accessed from anywhere
- BlazeMeter provides New Relic plug-in which helps in having monitoring eyes on the server. For this New Relic account (license) is required
- BlazeMeter uses Amazon cloud servers and all the test execution is done from the cloud
- BlazeMeter also provides plug-in support for Google Analytics, Apache JMeter, and Drupal
- While creating a new test, choose the Google Analytics option and BlazeMeter handles the rest. BlazeMeter analyzes the last 12 months of data and creates a test based on best practices
- This includes testing the top 5 most visited pages (which are directly accessible via HTTP) and calculating the number of concurrent users required and the delay between requests based on actual site usage data
- JMeter plug-in allows running the test from our workstation and exporting the execution data to BlazeMeter to analyze and generate the report
- BlazeMeter allows generating load by assigning Load Generator machines from different geographical locations. It too allows virtual users to use unique IP making it possible to simulate more realistic virtual users (IP Spoofing).
- BlazeMeter also supports few Continuous Integration tools like TeamCity, Jenkins, and JetBrains. Hence, it can be used in Agile mythologies
- It too provides Data security by masking the data. This is possible if rather than generating load from the Cloud we opt for on-premises load generation
- The most important and unique feature of BlazeMeter is its scripting engine. It uses JMeter as its scripting component
The key features of JMeter are:
- Multiple-Platform
- Highly Scalable
- Supports Multiple Protocols
- Availability of various types of Assertions and Listeners
- Good parameterization and correlation capabilities
Architecture of BlazeMeter
(Note: Click on any image for enlarged view)
- Here is the architecture of BlazeMeter. It has centralized BlazeMeter controller which is in Cloud. The test is controlled by Controller and depending on which option we opt for, the virtual users can be generated by Cloud or On-Premises Load Generator machines.
- These load generator machines create realistic mobile device users by simulating different devices and emulating a variety of network.
- The request then hit the application server which communicates with the database server. BlazeMeter collects the data from the application server and shows the results in graphical forms. If we have monitoring tool in place, it will also collect the performances counters and send it to the BlazeMeter.
- BlazeMeter integrates the monitoring data with test results and helps us in identifying the state of the server when we see performance degradation. Hence, help in identifying the performance issues.
BlazeMeter – FollowMe feature
FollowMe is the unique feature of BlazeMeter. It is being advertised by BlazeMeter with the punch line “No Scripting. No Recording. Just Testing“.
Here are few key features:
- Generate up to one million and even more virtual users to follow us in real-time.
- Virtual users mimic our browsing activity
- View real-time analytics & adjust our test on the fly
(Note: This graph is from BlazeMeter)
Here is the GUI of Follow me. This is Chrome plug-in and once installed can we see on the right-hand side of the browser.
Follow the number flow to run the test. Refer above figure in step 7 we can select the User Agent. I have taken the snapshot of the options coming in the drop-down for User Agents.
After hitting Start button (step-9), all the concurrent users will be ready to follow our steps. Whatever step we will perform on the browser, all of them will do it. On runtime, we can pause or stop the test. We can also see the test report and export the user activity to JMeter script.
BlazeMeter-Scenario Creation
This is the BlazeMeter scenario creation page. The load test scenario is designed from here. Under JMeter Test Properties section, JMeter scripts are uploaded. As BlazeMeter provides an option to run JMeter script of any version, we have to select JMeter version used to create the script.
If there is any command line argument, it can also be passed to the scenario. Through Host Override, Vusers is guided to generate load using unique IP address.
Network emulation section helps in configuring network settings. For better understanding, the snapshot of the same section is kept on the bottom right-hand side. From this section, we can select the network, set the bandwidth download limit and latency network delay. This helps in simulating real users.
If we have New Relic account, we can have monitoring eyes on the servers. Options New Relic APM and New Relic Insights are used for the same.
On the right-hand side, we have Load Scenario Properties section. In this section, we have “Location” option which helps in selecting the location from where we want to run the load test. Dropdown list shows the location of load generators in the cloud.
Then we have to configure no of concurrent users, ramp up, duration of the load test. Save the scenario and it`s ready to execute. In the trial version, we can run the test for half-an-hour and it supports the maximum of 50 Vusers.
In the above snapshot, we will get the idea about the test results and report. On the left-hand, we have scenario names on the upper side (Test) and test run for each scenario on the lower side (Session). Links above the graph are nothing but transaction names and available to select and deselect with the toggle option. The same toggle options with the user, response time and hit per sec.
On the drop-down at the right top, we can see different reports and logs we can get for the test. Reports and Logs could be Load Report, Aggregate report, Monitoring, Logs, Error logs, Original Test configuration.
BlazeMeter – Http URLs Test
HTTP URL test is a new scripting option. We have to keep adding the URLs and request that needs to send to those URLs. A JMeter script is automatically generated from the type of requests provided and during the execution it sends requests to the server sequentially from top to bottom.
As the generated script is similar to any manually created JMeter script, each Vuser follows the sequence of URL in the script and maintains its own browser session, cookies and cache. To render the HTML page as the response to the hit it sent to the server, it downloads all the resources on the web page including CSS files, js files, images and other static content.
In the snapshot above I tried to add 3 URLs of Times of India and these will be executed by a virtual user sequentially – first Home page then World page and then Business page.
We can add four types of requests i.e. GET, POST, PUT and DELETE. For post request, the parameters and their values that we need to pass the script can be identified using any network capturing tool like Fiddler.
BlazeMeter- Pros & Cons
Thank you so much for your valuable time to read the article. Hope this article is able to provide a good idea of the tool and reader will definitely give it a try.
In our next article, we will discuss more on Mobile App Security Testing Guidelines.
=> Visit Here For Complete Performance Testing Tutorials Series