Testing On Mobile Device – Part 2 – Network Data Mocking

Say, we have a perfect automation system,and we could basically control anything on the device, precisely. Would that imply we are going to have a perfect testing system right away?

Maybe not. Because data is also relevant.

For most automated testing systems, there is a common pattern:

1. actions / inputs to the system
2. assertions on the output
3. further actions
4. further assertions
5. ...

That's true for testing system for either web based testing system (e.g. selenium), or mobile applications (e.g. iOS automated testing).

If the system can be fully controlled, such as a standalone invoice system, it would be easy. However, for the other system, especially the mobile apps with dynamic data, that would not be the case.

For example, we have previously developed an iOS app for an online hiring company. And we have spent a lot of time to set up the Sikuli testing for the app. However, we found that there is no way for the testing to be successfully executed in the long run, because their data vary every day. And the worst part is, they actually show different UI for the different types of jobs in the search result.

So, how are we going to do such kinds of assertions?

We might do a lot of or conditions in the assert statements (not supported by Sikuli in nature but we could modify the source code if necessary), but that is still not an ideal solution. Maybe the particular UI is correct for today, but not for the other day. Using or operator would apparently make false positives.

Proposed Solution

1. We use WiredMock to record the all the network requests during Sikuli testing case set up.

2. Replay back the network data during Sikuli executions.

Some Complications

1. We don't want to modify the mobile app and change the URL inside, because as mentioned before, our ultimate goal is to minimize the work of engineers.

2. We use DNSMasq to fool the App to connect to the WiredMock server instead of the real application server. And WireMock itself would use another proxy server (supported by WireMock in nature) in order to escape from DNSMasq.

3. HAProxy is used in case we need to route multiple domains (of same App) into the WiredMock

So, here is our final solution:

We briefly tried it, and it indeed works.

In future we would try to integrate the data mocking with Sikuli and TightVNC, and hopefully could set up a complete testing system with minimum interference of engineers.