Mobile App Performance and Load testing

Blog: Capgemini CTO Blog

Many of our customers have correctly designed their infrastructure by sizing the servers and using design thinking to establish the mobile app’s UX – User Experience, but how can you tell whether your mobile app is going to work correctly in all the countries you want it to? What will be its real performance in countries that have poor internet capability or throughput? The problem is one of multi-country performance testing.

The Problem – Multi Country testing

So one option is to build the backend infrastructure and create a prototype app and simply functionally test the app manually from a written test plan. Of course, to test from all the countries in the world you would need fly the test guy between all the locations, or use staff in your various country offices. This however whilst telling you accurately that the App functionality works from country to country, is not an accurate way of telling how well the infrastructure will behave when under load. Nor will it tell you what the user experience is like when 10,000 users concurrently connect to your backend systems.

Another possibility is to use a well-known package such as HP’s (MicroFocus) Load runner or Storm runner with AppPulse Active. http://ow.ly/iJgY30j7x5I However, we found that the licence costs were prohibitive when you wanted to test 100 users from different countries, let alone 10,000 users. It might be easy to create the tests using Load runner, and the test developer support is first class, but the total cost of ownership is too high.

The third choice is to use an opensource performance and load testing tool, such as the Apache JMeter, which is in my opinion the most often used open source tool for load testing. Being open source this is effectively a free tool from a licence point of view. You still need to cost the writing of the tests and the analysing of the results, but you need to do that with whatever test strategy you use. But how do you test from multiple countries? – simples, you use the cloud.

For example In Capgemini we tend to choose one of the big three Cloud suppliers – Amazon AWS, Google Cloud and Microsoft Azure. Of these my personal preference is AWS Amazon Web Services because it supports in my opinion the greatest number of countries outlets around the world. Want to test in India, just spin up a EC2 linux M4.xlarge in Mumbai and then load Java and JMeter on it. Copy your tests across and start up a 10,000 user load test for an hour. When you finish, extract the performance results from the JMeter server and spin down the EC2. This is an effective, simple and a cheap form of load testing.

What is Apache JMeter

JMeter may be a 20 year old product, but it has been updated every year keeping up with the latest testing techniques. It is not a mobile app functional GUI tester, but instead concentrates on the communications protocol, so it tests the performance from the mobile through the back end systems API’s or services.

Many communication protocols can be served or supported by JMeter:

Within SAP the mobile communications are usually performed using OData from a SAP Gateway, the protocol with the Gateway typically uses HTTP or HTTPS and is therefore easily tested from JMeter. Therefore, we can add:

SAP OData Services via HTTP/HTTPS

It should however be noted that these performance tests and load tests do not test the mobile app functionally nor the mobile network WiFI/2G/3G etc. For that testing we use a different form of functional testing using Cucumber and Selenium, functional testing of mobile applications will be discussed in another blog. So why not use Cucumber for performance and load testing instead of JMeter ? Simply because of the loading of the server, you can run 20 concurrent functional tests on the same server you can run 10,000 JMeter load tests.

CI Continuous Integration

JMeter can easily be part of your Continuous Integration (CI) suite. CI is software development practice, where developers use version control system & integrate their code changes very often to the main branch and each change to the software is built and tested. The main idea behind the process is to detect any integration related issues as early as possible. This can be done by using the JMeter tests to functionally test the mobile services and performance test the infrastructure.

JMeter cen be supported from most of the popular CI suites for example: Jenkins, Maven and Gradle

One interesting use of load testing, is to load up the QC (quality) system when functional tests are taking place, this allows the QC system to appear as stressed as the production system, and this uncovers many potential problems.

On top of using load testing in QC, you can also use it against the production systems when out of normal working hours, to prove that the performance of the production system is the same as you expect.

Testing from Different Countries

As I mentioned earlier this is easily done by using the Cloud, imagine your top 3 primary mobile users are based in Australia, Canada and Ireland. How you going to load test that? Well you need to do what I explained before and spin up 3 cloud servers, one for each country and run the JMeter tests from each of these locations concurrently.

Advantages of using JMeter

Ease of Use: It is easy to install and use JMeter. Just download it from internet https://jmeter.apache.org/ , install and run. As a pure Java desktop application, it comes ready to use with default settings. It does not require you to have any specific skills or domain knowledge to use it. No special installation is needed. Everything you should do to run JMeter is to copy and run the *.bat file.

Open Source: JMeter is an open source software. This means that it can be downloaded free of cost. It is also a 100% pure Java application. If required the developer can even use its source code, can modify and customize it as per their requirements. They can also contribute their code to make an even better JMeter.

Platform independent: Jmeter is a platform independent application. On windows, testing on Jmeter can be initiated using the jmeter.bat file, and on Unix/Linux, by clicking on Jmeter Shell Scripts.

Robust Reporting: JMeter can generate the effective reporting. The test result can be visualized by using Graph, Chart, and Tree View. JMeter supports different formats for reporting like text, XML, HTML and JSON.

Ultimate Testing: With JMeter, user can do any kind of testing they want. Load Test, Stress Test, Functional Test, Distributed Test, all in one tool. It uses a multi-threading framework which is used for concurrent sampling using multiple threads.

Flexibility: You can customize JMeter as per your requirement and apply the automation testing to JMeter. You can save the effort of executing test cases manually.

Multi Protocol Support: JMeter supports several protocols like HTTP, FTP, SOAP, JDBC, JMS, and LDAP. It can also be used for testing the performance of your database.

Test Format: JMeter creates and stores its test plan in XML format. That indicates that we can generate or create a test plan using a text editor.

Friendly GUI (Graphic User Interface). Actually, it could be used in 3 modes: GUI Mode, Server Mode, and Command Line Mode. It takes a little time to get familiar with JMeter GUI.

Test case result visualization. The test result can be displayed in different ways: table, chart, log file and tree.

Record and Playback options. If required, the user activity on the browser could be recorded and simulated in the web application using JMeter.

Integrated real-time support. Apache Tomcat collectors allow providers for real-time monitoring.

Disadvantages of using JMeter

Memory Consumption: JMeter can simulate heavy load and visualize the test report. This may consume lots of memory and can lead out of memory under heavy load. If using AWS, then best to specify a EC2 linux M4.xlarge system.

Web application only: JMeter is good tool for testing web application but it is not a suitable tool for testing desktop application. In Capgemini we tend to use Cucumber/Selenium based tests for functionally testing mobile or web applications.

Lack of support for JavaScript: JMeter is not a browser, so it cannot run JavaScript in the web application. It has limited support for processing JavaScript or Ajax, this may affect the accuracy of simulation.

Recording HTTPS: Currently there is no ability to record HTTPS communication, but you can use HTTPS by manually creating the tests.

Ajax traffic: Currently there is no option of the AJAX traffic interception.

Conclusions on using JMeter tests from multiple countries.

Identify potential bottlenecks of your mobile application
Discover the number of concurrent users that can access your mobile application without a heavy degradation of the user experience
Find out the breaking points of the technology stack used by the mobile application
Find out how slow the server APIs or any other web service respond when under heavy load
Understand if the database technology represent a bottleneck in your mobile application
Find out if changes done in each mobile or service release had a bad influence on the performance behaviour of the mobile application
Fine tuning the different technology stack components
Know how many parallel users your mobile application services can handle