Faster Gradle builds through caching and parallelization

Spend less time waiting for your build to run and more time developing software by enhancing your Gradle build with:

1. Caching
2. Parallel execution

Caching

The fastest part of your build is one you don't even need to run at all.

Using the Gradle build cache can save Gradle from re-running work that was already run elsewhere. For example, avoid re-running tasks (compilation, tests, etc.) that were already run on your CI server. With the Gradle build cache, Gradle will pull down those results as part of your local build (or another CI build) and saving you a ton of time by avoiding unnecessarily re-running those tasks.

To enable the build cache in your projects add this property to your gradle.properties file:

1org.gradle.caching=true

Now that your project has the build cache enabled, the next step for the most savings it to set up a remote build cache backend.

Remote build cache savings

Gradle supports both local and remote build caches. While using only the local build cache can help save you time during certain operations like switching between branches locally, the biggest time savings will come with the remote build cache.

With the remote build cache, you can re-use the work done by your CI machines across all the rest of your builds - from local developer builds to other CI builds as well.

For example, say you have a multi-project build and when you start work in the morning you pull the latest code from the repo. Then as part of your day's work you only change one subproject in the build. Without the build cache, Gradle will also have to compile, run tests, etc. in each project that was changed by you or anyone else since you last ran your build locally.

But with the build cache, Gradle can re-use the output - such as compiled code, test executions, etc. - from CI builds. And you don't have to unnecessarily re-run tasks for projects other than the single subproject you changed locally. Letting you leverage the work all your CI builds already performed.

I've saved many, many hours of build time just on my open source projects by using the remote build cache to avoid re-running extensive test suites in subprojects that didn't change.

Remote build cache configuration

To use the Gradle remote build cache, you'll need a remote build cache backend. There are a variety of options you can use for a remote cache backend. For example, Gradle supports an HTTP backend out of the box. If you already have an Artifactory server, you can use that as a remote cache backend.

And there a variety of open-source remote build cache configuration options as well. For example, I wrote a remote cache Gradle plugin for any S3-compliant storage solution and I personally use it with Digital Ocean Spaces to save a ton of time building my open source projects.

Parallelization

And for the parts of your build you still do need to execute, you can also shorten the overall duration of your build by running portions of it in parallel.

There are two main ways to parallelize your build:

1. Building subprojects in parallel
2. Running tests in parallel

Parallel subprojects

You can speed up the overall duration of a multi-project build by having Gradle run independent subprojects in parallel.

For example, say we have a multi-project build with the following subprojects:

• :foo-client
• :foo-notifications
• :foo-server
• :foo-utils

In this example project, say the :foo-server subproject depends on :foo-client, :foo-notifications, and :foo-utils. Those last three subprojects are independent of each other, so Gradle can safely run their tasks (compilation, test execution, etc.) in parallel. And then run the tasks :foo-server once they all complete - saving you time in the overall duration of your build vs running each project one after the other.

Set the following property in your gradle.properties file to enable parallel project execution:

1org.gradle.parallel=true

Now Gradle will analyze which projects can be safely run in parallel based on the project interdependencies - and save you time by running those projects in parallel.

Parallel test execution

In addition to running projects in parallel, you can speed up your build by running tests in parallel in an individual project. This can speed up the overall test execution duration of that project.

For example, here's a test run with the tests running serially - one after the other - taking 4m 13s:

And then the same set of tests run in parallel - taking only 1m 3s:

In the trace of the parallel test run, you can see 4 test cases running in parallel.

Enabling parallel tests

To enable parallel test execution in your project, set the maxParallelForks parameter in your build.gradle file:

1test {
2    maxParallelForks = Math.floorDiv(Runtime.runtime.availableProcessors(), 2) + 1
3}

Now Gradle will fork the test execution into set of threads based the number of processors on the machine running the build.

Independent parallel tests

One key element in successful parallel testing is that tests are independent of each other. For example, tests that read and/or write to the database can't interact with each other's data. Otherwise, the tests could conflict with data from others tests and fail intermittently when run in parallel.

When tests read/write from a database, there are different ways you can separate their data interactions so you can safely run them in parallel. For example, you could:

1. Generate random values for key test data (e.g. IDs) so each test's data is unique to that test case
2. Run each test case inside a database transaction that the database changes are rolled back when the test case finishes

With either of those approaches, you can separate tests from interacting with each other's data when run in parallel and avoid annoying intermittent failures from parallel test runs.

Conclusion

By using the Gradle build cache, parallel project execution, and parallel test execution - you can spend less time waiting for your build and more time writing valuable software.