Hamcrest, JaCoCo, Mockito and Jenkins for Android

JUnit on its own may be all you need but there are a number of excellent third party tools that you can bolt onto JUnit that really make your Android testing shine.

In this post we’ll take a look at the following tools:

  • Hamcrest for better assertions
  • JaCoCo so we can measure our JUnit code coverage
  • Mockito so we can keep our unit tests focused on the code
  • Jenkins for automating our testing

Hamcrest Assertions

Anything more than a simple Hello, World type application is probably going to need better assertions than what comes with jUnit 4.x assertions. Hamcrest is one option that offers a lot more matchers. It also provides a lot more flexibility by being allowing you to now include ranges instead of just single values. As the Hamcrest documentation says, Hamcrest lets you create “Matchers that can be combined to create flexible expressions of intent”. Table 1 has a list of most of the Hamcrest Assertions available and you can also write your own.

Package Assertions
CoreMatchers allOf, any, anyOf, anything, array, both, containsString, describedAs, either,endsWith, equalTo, everyItem, hasItem, hasItems,,instanceOf, is, isA, not, notNullValue, nullValue, sameInstance, startsWith, theInstance
Matchers allOf, any, anyOf, anything, array, arrayContaining, arrayContainingInAnyOrder,arrayWithSize, both, closeTo, comparesEqualTo, contains, containsInAnyOrder, containsString, describedAs, either, empty, emptyArray, emptyCollectionOf, emptyIterable, emptyIterableOf, endsWith, equalTo, equalToIgnoringCase, equaltToIgnoringWhiteSpace, eventFrom, everyItem, greaterThan, greaterThanOrEqualTo, hasItem, hasItemInArray, hasItems, hasKey, hasProperty, hasSize, hasToString, hasValue, hasXPath, instanceOf, is, isA,isEmptyOrNullString, isIn, isOneOf, iterableWithSize, lessThan, lessThanOrEqualTo, not, notNullValue, nullValue, sameInstance, samePropertyValueAs, startsWith, stringContainsInOrder, theInstance, typeCompatibleWith
Condition and, matched, matching, notMatched, then
MatcherAssert assertThat
Table 1. Hamcrest Assertions.

Listing 1 shows how to add the Hamcrest library to your build.gradle file to include Hamcrest functionality into your app. Remember to hit the Sync now button.

dependencies {
    testCompile 'junit:junit:4.12'
    testCompile 'org.hamcrest:hamcrest-library:1.3'
}
Listing 1. Adding Hamcrest library dependency.

Now we refactor our tests so they read more like English, see Listing 2

@Test
public void calculator_CorrectHamAdd_ReturnsTrue() {
    assertThat(mCalculator.add(3, 4), is(7));
}
Listing 2. Hamcrest assertions.

We can also add ranges to our tests using greaterThan and LessThan assertions, see Listing 3.

public void calculator_CorrectHamAdd_ReturnsTrue() {
    assertThat(mCalculator.add(3, 4), greaterThan(6));
    assertThat(mCalculator.add(3, 4), lessThan(8));
}
Listing 3. greaterThan and lessThan.

Or we can combine the two using the both command, see Listing 4.

@Test
public void calculator_CorrectHamAdd_ReturnsTrue() {
    assertThat(mCalculator.add(3, 4), both(greaterThan(6)).and(lessThan(8)));
}
Listing 4. Using the both matcher.

JaCoCo

Unit testing needs some form of code coverage to find any untested parts of the code. Code coverage tools output code metric reports and annotated code to show just what code has been unit tested (in green) and what has not been covered by a unit test (in red). Figure 1 shows the code coverage figures for JaCoCo which was taken from the eclemma.org website.

JaCoCo Coverage

Figure 1. Code Coverage example.

The Code Coverage metric measures how much source code has been unit tested. Personally I’m not a huge believer in having a code coverage metric target on an Android project, it should be used as a guide rather than a mandated requirement. However if a project has 5% code coverage then you’re not really doing unit testing, and are only paying lip service to the technique.

Android Studio will invoke or call JaCoCo to do the code coverage reports on your unit tests but you need to perform the following tasks.

  • Set testCoverageEnabled to true in the build.gradle file
  • Change the Code Coverage Runner to JaCoCo
  • Run Unit Tests with Code Coverage
  • View the Code Coverage

To include code coverage in your Android project, set testCoverageEnabled to true in your debug buildTypes in the build.gradle file, see Listing 5, and click on Sync Now after you make the changes.

buildTypes {
    debug {
        testCoverageEnabled true
    }
}
Listing 5. build.gradle JaCoCo changes

To edit the configurations, go to Run->Edit Configurations, see Figure 2.

Edit Configurations

Figure 2. Choose Edit Configurations

Click on the Code Coverage tab and change the coverage runner to JaCoCo, see Figure 3.

Changing coverage runner

Figure 3. Changing coverage runner

Run the tests now by right clicking on the method and choosing Run CalculatorTest with Coverage, see
Figure 4.

Run Unit Test

Figure 4. Run Unit Test with Coverage

The code coverage reports are showing in the Coverage tab, see Figure 5, where you can see we have
50% code coverage in our Calculator method.

Code Coverage Tests

Figure 5. Code Coverage tests.

The code coverage red/green is show in the method, although it can be hard to see, see Figure 6. The JaCoCo integration in Android Studio is new. No doubt future versions will use much easier to see red/green coverage.

Code Coverage

Figure 6. Code Coverage.

Mockito

In the last post in the Grouping tests section we talked about Small, Medium and Large tests. In reality unit tests should always be small tests. But if we’re making network connections or reading from the file system or database then by definition we’re not performing small unit tests. We are also making an assumption about a third party web service or database that may not be running every time we run our tests. So worse case scenario our tests are going to fail, but for the wrong reason such as the network being down. We use mocking frameworks to mock out any code that talks to external resources and get all of our unit tests back to the smaller group. Mockito works very well with Android Studio so we’re going to use that in this and subsequent posts.

Listing 6 shows how to add the Mockito library to your build.gradle file by including the testCompile 'org.mockito:mockito-core:1.10.19' library. Once again remember to hit the Sync now link after you’re done.

dependencies {
    testCompile 'junit:junit:4.12'
    testCompile 'org.hamcrest:hamcrest-library:1.3'
    testCompile 'org.mockito:mockito-core:1.10.19'
}
Listing 6. Adding mockito library.

Google’s Android sample has a networking app called NetworkConnect which you can find at
https://github.com/googlesamples/android-NetworkConnect. Figure 7 shows the basic functionality of the app which returns the HTML for the Google webpage.

NetworkConnect app

Figure 7. NetworkConnect app.

Before we mock out the code we’re going to need to cut and paste the network access code into its own class, see Listing 7, which we’ll call DownloadUrl.

public class DownloadUrl {

    public String loadFromNetwork(String urlString) throws IOException {
        InputStream stream = null;
        String str ="";

        try {
            stream = downloadUrl(urlString);
            str = readIt(stream, 88);
        } finally {
            if (stream != null) {
                stream.close();
            }
        }
        return str;
    }

    public InputStream downloadUrl(String urlString) throws IOException {
        URL url = new URL(urlString);
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        conn.setReadTimeout(10000 /* milliseconds */);
        conn.setConnectTimeout(15000 /* milliseconds */);
        conn.setRequestMethod("GET");
        conn.setDoInput(true);
        conn.connect();
        InputStream stream = conn.getInputStream();
        return stream;
    }

    public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
        Reader reader = null;
        reader = new InputStreamReader(stream, "UTF-8");
        char[] buffer = new char[len];
        reader.read(buffer);
        return new String(buffer);
    }
}
Listing 7. DownloadUrl code.

The MainActivity now calls DownloadUrl as follows, see Listing 8.

private class DownloadTask extends AsyncTask<String, Void, String> {
    DownloadUrl htmlStr = new DownloadUrl();

    @Override
    protected String doInBackground(String... urls) {
        try {
            return htmlStr.loadFromNetwork(urls[0]);
        } catch (IOException e) {
          return getString(R.string.connection_error);
        }
    }

    /**
     * Uses the logging framework to display the output of the fetch
     * operation in the log fragment.
     */

    @Override
    protected void onPostExecute(String result) {
      Log.i(TAG, result);
    }
}
Listing 8. Updated NetworkConnect MainActivity code.

We can now write a unit test to see if the DownloadUrl code is returning HTML in our unit test, see Listing 9.

public class DownloadUrlTest {
    DownloadUrl tDownloadUrl;
    String htmlStr;

    @Before
    public void setUp() {
        try {
            htmlStr = tDownloadUrl.loadFromNetwork("http://www.google.com");
        } catch (IOException e) {
            // network error
        }
    }

    @Test
    public void downloadUrlTest_ReturnsTrue() {
            assertThat(htmlStr,containsString("doctype"));
    }
}
Listing 9. Network Connect unit test.

Because we’re making a network call, we should mock out the network access using Mockito. For this example there are only a couple things we need to do to mock out the webserver call. First mock out the class so Mockito knows what functionality it needs to replace DownloadUrl tDownloadUrl = Mockito.mock(DownloadUrl.class); Next tell Mockito what to return when the method you’re testing is called using the Mockito.when().thenReturn() format which is as follows Mockito.when(tDownloadUrl.loadFromNetwork("http://www.google.com")).thenReturn("<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"en\"><head>"); Now when the loadFromNetwork call is made it will return our partial webpage instead of the much actual HTML of http://www.google.com webpage, see Listing 10. You can test this by turning on and off your network access.

@RunWith(MockitoJUnitRunner.class)
public class DownloadUrlTest {
    public DownloadUrl tDownloadUrl = Mockito.mock(DownloadUrl.class);

    @Before
    public void setUp() {
        try {
            Mockito.when(tDownloadUrl.loadFromNetwork("http://www.google.com")).thenReturn("<!doctype html><html itemscope=\"\" itemtype=\"http://schema.org/WebPage\" lang=\"en\">");
        } catch (IOException e) {
            // network error
        }
    }

    @Test
    public void downloadUrlTest_ReturnsTrue() {
        try {
            assertThat(tDownloadUrl.loadFromNetwork("http://www.google.com"),containsString("doctype"));
        } catch (IOException e) {
            //
        }
    }
}
Listing 10. Mocked Network Access.

We will return to Mockito in the next post and show how to mock out database and shared preferences access as well as how to use other tools to extend the Mockito functionality to help decouple or separate out your code.

Jenkins

Moving to an Agile process can create considerable overhead. Thankfully we no longer have to worry about the emulator taking so long to fire up for vanilla junit tests. It takes seconds now instead of minutes. However as the app grows and the corresponding number of unit tests grows too, then eventually it’s going to take time to run the tests manually. The number of steps to build and test the apps correctly will also start to become more complex. And as humans are not good at tedious multi step processes so it makes sense to use a Continuous Integration (CI) server to automate the process wherever possible to reduce any unnecessary testing errors.

For our purposes we’re going to use Jenkins because it has so many plugins available. However there are many other options such as Travis, TeamCity or Bamboo that can work equally well if you’re more familiar with those CI environments.

Install

Download the Jenkins server from http://jenkins-ci.org/. Install it and go to http://localhost:8080 and you should see figure 8.

Jenkins Start Screen

Figure 8. Jenkins start up screen.

Configure Jenkins

To make it useful in our Android Environment we’re going to need to add a number of plugins. Click on Manage Jenkins->Manage Plugins and search for and add the Gradle and GIT Plugin or whatever other source code management system you use. When you’re done your installed plugins screen should look something like Figure 9.

Installed plugins

Figure 9. Installed plugins.

Next we need to configure Jenkins so it knows where you installed Android. Click on Manage Jenkins->Configure System, scroll down to Global Properties, click on Environment variables checkbox and enter the directory for the ANDROID_HOME where you installed Android see Figure 10.

Environment Variables

Figure 10. Setting Environment Variables.

Create Automated job

Now that we’ve configured Jenkins we need to create our first automated job. Go back to the Dashboard and click on create a new job. Enter the name of your project and Choose Freestyle Project, see Figure 11.

Create New Item

Figure 11. Create New Item.

We need to tell Jenkins, where to find the code. In this example we’re using Git as our source code management system. Here we’re again using the Google NetworkConnect sample. Enter the Git repository URL. As it’s a public repo there are no credentials so we’re going to skip that. There is also only one branch so we can leave the Branch Specifier as master, see Figure 12.

Network Connect repository

Figure 12. Enter Network Connect repository details.

Scroll down to the Build section and choose Invoke Gradle script, see Figure 13.

Gradle Script

Figure 13. Invoke Gradle Script.

In the Build step choose Use Gradle Wrapper, check Make gradlew executable and From Root Build Script Dir. Enter –refresh-dependencies and –profile in the switches section. And in this case enter assemble in the Tasks section. Click save, see Figure 14.

Configure Build

Figure 14. Configure the Build.

Now we’re ready to build our app. Click on Build Now on the Project page, see Figure 15.

Project Page

Figure 15. Project Page.

Once the build starts you’ll see a progress indicator to see how your task is doing. If you want to see what’s happening then click on the Build Number, see Figure 16.

Build Progress

Figure 16. View Build Progress.

Now click on the Console Output and you can see what’s happening as if you were running the app from the command line, see Figure 17.

Console Output

Figure 17. Click on Console Output.

In our example there are no errors and the build is successful, see Figure 18. If that’s not the case then the Console Output page can be really helpful to see what failed.

View Console Output

Figure 18. Console Output.

We’ll be using Jenkins later to automate our jUnit and Espresso tests.

Summary

In this post we’ve looked at a number of tools that we’re going to use throughout the blog to make our testing more effective and more efficient. In the recent past it’s been a very frustrating task to get this stack up and running, but thankfully that is no longer the case.

Note a complete set of these testing blogs have been compiled together into an Agile Android mini-book, published by Apress and available from Amazon

This is Part 3 of 6, the remaining blogs can be found below.

Part 1 - Android Testing
Part 2 - Android Unit Testing
Part 3 - Hamcrest, JaCoCo, Mockito, and Jenkins for Android
Part 4 - Espresso Testing on Android
Part 5 - Android Mocking
Part 6 - Maintaining Someone Else’s Android Code