Testing On Mobile Device – Part 1 – Sikuli – VNC

Facts

1. We are a small sized IT company, with around 50 staff and 30 developers.

2. Like most other IT companies in the world, we don't have full-time engineers work as QA.

3. Most engineers got more one project at a time. (One for development and one for maintenance.)

4. Engineers don't have the time (or they just don't care) to write the unit tests and functional tests, even if the tools are provided by iOS and Android development frameworks.

5. Our QA did most of the testing manually, but it's a boring task. And they are not capable of writing the codes for the unit tests and functional tests.

6. The Apps for iOS simulator and device have significantly different architecture, but the apps for Android simulator and device are the same.

7. If the tasks depend on the work of engineers, it normally means the tasks need cooperation of engineer and QA, and the timing would be a big problem.**

So, our goal is

Allow our QA to set up the automated tests for the mobile applications, with zero or minimum efforts of engineering.

---

Solution A

iOS Side:

1. Engineer builds and runs the application on the simulator in his development machine.

2. Engineer copy the simulator app folder to the testing machine

3. Use shell script to launch the simulator

4. Use sikuli to interact with the app of simulators (e.g. click the device or scroll the screen). Note that we have modified sikuli source code a little bit to make sure there is a little bit delay between each action.

5. After sikuli finished running, generate the result file.

6. Use script to parse the result file, and report success or not

7. Step 3 - 6 are automatically triggered by Jenkins, and the final result is reported back Jenkins as the result.

Android Side:

1. Engineers builds and archive the application (APK) on the development machine

2. Engineer copy the APK binary to a specific location of testing machine

3. Use shell script to install binary into the emulator using ADB

4. Use shell script to launch the application using ADB

5. Do the same steps 5-7 as iOS side

Advantage

1. UI of simulator/emulator is smooth, and responsive

2. 90% of work could be automated

Disadvantage

1. The biggest disadvantage is, the engineers need to prepare the binary for the testing machine. Though the effort is small, but if the release frequency is high, it would still cause a lot of overhead for the engineers. It seems odd, but it's just not working in reality after a while.

2. Some apps could not work on the simulator, it only works for the devices. For example, the in-app purchase of iOS, or some video rendering logic.

---

Solution B (We are currently working on)

iOS Side:

1. Jailbreak an iOS device

2. Install veency on the device

3. Use shell script to connect to the iOS device using tightVNC java viewer. (The source code needs to be modified a little bit to get started without any prompt.)

4. Use Sikuli to interact with the vnc viewer, and hence interact with the device in return.

5. After sikuli finished running, generate the result file.

6. Use script to parse the result file, and report success or not

7. Step 3 - 5 are automatically triggered by Jenkins, and the final result is reported back Jenkins as the result.

Android Side

1. root the Android Device

2. Install Droid VNC on the android machine

3. Do the same steps 3-7 as iOS side

Advantage

1. The tester would download the device from the release link that's sent to the customer. The engineer does not need to do any extra work.

2. We are using the REAL device for the testing, so basically we could do anything the device can (e.g. purchase items).

Disadvantage

1. Slow, the performance of VNC is not so good, because the very high resolution of devices. Even if we only used intranet for the communication, it's still slow.

2. Due to the limitation of VNC, some items could not be viewed, e.g. the videos. But on the other hand, we are not going to do automated testing of videos anyway.

In general, the two disadvantages are relatively mild.

---

One last issue

Another problem we hit previously is in many cases the mobile device would try to get the data from the network, and that causes big problems for automated testing.

Using either Sikuli, Selenium or even SDK related automation framework, all functional tests try to open some screens and search some keywords. But it's hard if the data changes all the time. And what's worse is, sometimes the UI is related to the data.

For example, we have once set up an automated testing for a job-seeking company. However, for different job titles, a different screen would be shown. And due to the nature of job seeking company, the data basically change every single day.

We would discuss the possible solutions in the next post.