Building Better UI Tests with AI

When developing websites and apps, UI tests are written to ensure the products delivered are of high quality and that defects are quickly identified and fixed should they come up. Although they are extremely useful, UI testing can be tedious work as tests can be brittle to changing app requirements, and every new app requires a new set of UI tests. But what if we could write a test, powered by AI that is robust to app changes, as well as generalizes to new apps without changing the test at all.

As mentioned previously, UI tests can be very fragile. They can break if any of the elements on the screen are renamed, removed, replaced, or moved. In an attempt to combat some of these failings, Test.ai developed an image classifier (utilizes AI to find similar images) to semantically search for elements in your app, rather than by id. What does this mean? It means that rather than using one of the ordinary strategies for finding elements on your app which include searching by, Accessibility ID, Class Name, ID, Name, XPath or Image, we can search for images and icons that match the “idea” of what we are looking for. For example, we could say “click the search button”, or “find the shopping cart icon” without explicitly telling the test what we are looking for. If there is an image that looks similar to search button or cart icon on the screen, we will find it automatically. A list of what can be searched for with this tool can be found here.

Now that we can just tell the UI test what we are looking for, this test will work unchanged on similar apps. Ideally, in the future we will just be able to tell our computer “test the sign in feature” and be able to test any app requiring login. But we aren’t there yet.

The key technologies we will use for this demo are:

  • Genymotion
  • Appium
  • Test.ai image classifier plugin for Appium

Prerequisites

Before we start, make sure you have done the following:

  • Installed Python 3
  • Installed Node.js package manager
  • Installed Android Studio
  • Made sure Android SDK and adb have been added to path.

The directions for this demo are specifically for MacOS users, but most of the instructions will apply to any OS.

Setting up your environment

To emulate our Android environment we will need to download and install Genymotion. Once you have Installed Genymotion on your machine, set up an Android device. We will setup a Google Pixel, but feel free to use any device you are comfortable with.

Next, we need to install Appium. To do this, open up the terminal and execute the following command:

npm install -g appium

This will install the Appium module globally on your machine. If the above fails, run the following commands and then retry:
sudo chown -R $(whoami) ~/.npm
sudo chown -R $(whoami) /usr/local/lib
sudo chown -R $(whoami) /usr/local/bin

Next we will install the dependencies required for Test.ai’s classifier plugin.

brew install pkg-config cairo pango libpng jpeg giflib

Then we will need to install the Test.ai module in the global Appium directory we created earlier.
cd /usr/local/lib/node_modules/appium
npm install test-ai-classifier --save

Defining our problem

For this demo, we will write a test that will work on 2 different movie apps. We want to ensure that search functionality works on each of these apps. We will use AI to automatically find the search button with its image classifier, enter in some text, then search for a movie. This test will be robust enough to where we can change the id name, location or even the image asset for our search button, and the test will not break. As well as have it perform well on our two apps as showed below.

As we can see, the search buttons on each app are:

  • Have different locations
  • Have different image assets
  • Have different color background

What we can not see but in the images but also holds true is that the two search buttons are different types of objects and have different ID. No ordinary test would be able to find the search button on each of these apps without changing the test. Luckily for us, we have AI.

Writing your test

Next, we will need to write our Appium test. We will do this in Python, but other languages such as Java are supported. First, we will Install the python unit testing framework and the python Appium client.
pip3 install pytest Appium-python-client

Now we can write our script. We will create one test which will find the search button automatically with an image classifier, enter some text into the text field, and then search for the movies.

Note: if you are following along, you will need to find the IP address of your genymotion emulator with the following command
adb devices

Running your test

1. Open Genymotion and start up your Android Emulator
2. Open the terminal and run the command:
appium
3. Start up a new tab on terminal and run the following commands
python3 test.py /my_directory/myapp.apk

We will see that the test.py script will then install our movie app on the Genymotion emulator and will click the search button on the top right corner of the screen. Enter in our movie name and select a result. It’s as easy as that.

Related Posts

  • Was this Helpful ?
  • Yes   No