Recently I was preparing tests before releasing Booster v3.0.0. To ensure Booster‘s quality, I adapted it to every version of the Android Gradle Plugin from 3.0.0 to 4.1.0 and wrote a large number of integration tests. While running the test cases, the debug build tests went smoothly at first. Then on a whim I decided to add release builds too. To my surprise, the test suite got stuck on the Android Gradle Plugin 3.5.0 case and wouldn’t budge. After many retries with the same result, I checked and found it was already OOM – a pile of hprof files had been dumped. Looking at the JUnit test report, I discovered it was a Metaspace overflow.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
org.gradle.testkit.runner.UnexpectedBuildFailure: Unexpected build execution failure in /var/folders/dy/bxzw01fx3bv9xmsmyy77qgx80000gp/T/junit1018081279555935171 with arguments [assemble, -S, -Pbooster_version=2.4.0, -Pandroid_gradle_version=3.5.0, -Pcompile_sdk_version=28, -Pbuild_tools_version=26.0.3, -Pmin_sdk_version=18, -Ptarget_sdk_version=26]

Output:
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:processResources
> Task :buildSrc:classes
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build

> Configure project :
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/java/main/
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/kotlin/main/
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/tmp/kapt3/classes/main/
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/resources/main
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-compat/build/libs/booster-android-gradle-compat-3.0.0-SNAPSHOT.jar
file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.3.61/2e07c9a84c9e118efb70eede7e579fd663932122/kotlin-reflect-1.3.61.jar
file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.3.61/4702105e97f7396ae41b113fdbdc180ec1eb1e36/kotlin-stdlib-1.3.61.jar
file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.3.61/65abb71d5afb850b68be03987b08e2c864ca3110/kotlin-stdlib-common-1.3.61.jar
file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/kotlin/integrationTest/
WARNING: The specified Android SDK Build Tools version (26.0.3) is ignored, as it is below the minimum supported version (28.0.3) for Android Gradle Plugin 3.5.0.
Android SDK Build Tools 28.0.3 will be used.
To suppress this warning, remove "buildToolsVersion '26.0.3'" from your build.gradle file, as each version of the Android Gradle Plugin now has a default version of the build tools.

> Task :preBuild UP-TO-DATE
> Task :preCnDebugBuild UP-TO-DATE
> Task :compileCnDebugAidl NO-SOURCE
> Task :compileCnDebugRenderscript NO-SOURCE
> Task :checkCnDebugManifest
> Task :generateCnDebugBuildConfig
> Task :mainApkListPersistenceCnDebug
> Task :generateCnDebugResValues
> Task :generateCnDebugResources
> Task :createCnDebugCompatibleScreenManifests
> Task :processCnDebugManifest
> Task :mergeCnDebugShaders
> Task :compileCnDebugShaders
> Task :generateCnDebugAssets
> Task :mergeCnDebugAssets
> Task :processCnDebugJavaRes NO-SOURCE
> Task :checkCnDebugDuplicateClasses
> Task :validateSigningCnDebug
> Task :signingConfigWriterCnDebug
> Task :mergeCnDebugJniLibFolders
> Task :preCnReleaseBuild UP-TO-DATE
> Task :compileCnReleaseAidl NO-SOURCE
> Task :compileCnReleaseRenderscript NO-SOURCE
> Task :checkCnReleaseManifest
> Task :generateCnReleaseBuildConfig
> Task :mainApkListPersistenceCnRelease
> Task :generateCnReleaseResValues
> Task :generateCnReleaseResources
> Task :javaPreCompileCnRelease
> Task :createCnReleaseCompatibleScreenManifests
> Task :processCnReleaseManifest
> Task :prepareLintJar
> Task :mergeCnReleaseShaders
> Task :compileCnReleaseShaders
> Task :generateCnReleaseAssets
> Task :mergeCnReleaseAssets
> Task :processCnReleaseJavaRes NO-SOURCE
> Task :checkCnReleaseDuplicateClasses
> Task :signingConfigWriterCnRelease
> Task :mergeCnReleaseJniLibFolders
> Task :preEnDebugBuild UP-TO-DATE
> Task :compileEnDebugAidl NO-SOURCE
> Task :compileEnDebugRenderscript NO-SOURCE
> Task :checkEnDebugManifest
> Task :generateEnDebugBuildConfig
> Task :mergeCnDebugResources
> Task :processCnDebugResources
> Task :mainApkListPersistenceEnDebug
> Task :generateEnDebugResValues
> Task :generateEnDebugResources
> Task :mergeCnDebugNativeLibs
> Task :stripCnDebugDebugSymbols
> Task :createEnDebugCompatibleScreenManifests
> Task :processEnDebugManifest
> Task :checkEnDebugDuplicateClasses
> Task :mergeEnDebugShaders
> Task :compileEnDebugShaders
> Task :generateEnDebugAssets
> Task :mergeEnDebugAssets
> Task :processEnDebugJavaRes NO-SOURCE
> Task :validateSigningEnDebug
> Task :signingConfigWriterEnDebug
> Task :mergeEnDebugJniLibFolders
> Task :javaPreCompileCnDebug
> Task :compileCnDebugJavaWithJavac
> Task :compileCnDebugSources
> Task :transformClassesWithDexBuilderForCnDebug FAILED
> Task :mergeCnReleaseNativeLibs
> Task :mergeCnDebugJavaResource
> Task :mergeEnDebugNativeLibs
> Task :mergeCnReleaseResources
> Task :javaPreCompileEnDebug
> Task :mergeEnDebugResources
Daemon will be stopped at the end of the build after running out of JVM memory

FAILURE: Build failed with an exception.

* What went wrong:
Metaspace

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
java.lang.OutOfMemoryError: Metaspace


* Get more help at https://help.gradle.org

BUILD FAILED in 24s

at org.gradle.testkit.runner.internal.DefaultGradleRunner$2.execute(DefaultGradleRunner.java:255)
at org.gradle.testkit.runner.internal.DefaultGradleRunner$2.execute(DefaultGradleRunner.java:251)
at org.gradle.testkit.runner.internal.DefaultGradleRunner.run(DefaultGradleRunner.java:324)
at org.gradle.testkit.runner.internal.DefaultGradleRunner.build(DefaultGradleRunner.java:251)
at io.bootstage.testkit.gradle.rules.GradleExecutor.finished(GradleExecutor.kt:50)
at org.junit.rules.TestWatcher.finishedQuietly(TestWatcher.java:117)
at org.junit.rules.TestWatcher.access$400(TestWatcher.java:46)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:64)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.lang.Thread.run(Thread.java:748)

Gradle Testing

Running Gradle plugin tests requires Gradle TestKit. Before each @Test method, a temporary Android Gradle project is dynamically created, and after the @Test method, the Gradle Runner executes the assemble task. Within the @Test method body, you can manipulate this dynamically generated Android Gradle project. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
private const val MIN_SDK_VERSION = 18
private const val TARGET_SDK_VERSION = 30
private val AGP = V30

private val ARGS = arrayOf(
"assemble", "-S",
"-Pbooster_version=${Build.VERSION}",
"-Pandroid_gradle_version=3.0.0",
"-Pcompile_sdk_version=28",
"-Pbuild_tools_version=26.0.3",
"-Pmin_sdk_version=${MIN_SDK_VERSION}",
"-Ptarget_sdk_version=${TARGET_SDK_VERSION}"
)

@Suppress("RemoveCurlyBracesFromTemplate", "FunctionName")
abstract class V30IntegrationTest(val isLib: Boolean) {

private val projectDir = TemporaryFolder()

@get:Rule
val ruleChain: TestRule = rule(projectDir) {
rule(LocalProperties(projectDir::getRoot)) {
rule(TestCaseConfigure(projectDir::getRoot)) {
GradleExecutor(projectDir::getRoot, "4.1", *ARGS)
}
}
}

@Before
fun setup() {
projectDir.copyFromResource("${if (isLib) "lib" else "app"}.gradle", "build.gradle")
projectDir.copyFromResource("buildSrc")
projectDir.copyFromResource("src")
}

@Test
@Case(ScopeFullWithFeaturesTest::class)
fun `test AGPInterface#scopeFullWithFeatures`() {
}

}

class V30AppIntegrationTest : V30IntegrationTest(false)

class V30LibIntegrationTest : V30IntegrationTest(true)

Gradle Runner

When the Gradle Runner executes a test build task, it spawns a new GradleDaemon process. By default, the Gradle Runner creates a directory similar to ~/.gradle under $TMPDIR for caching files. When the OOM first appeared, I suspected it was related to the Gradle Runner, but after resetting the Gradle Runner‘s directory, the OOM persisted.

Profiling

Since the OOM occurred in Metaspace, it was most likely related to class loading. So I used VisualVM to profile the Gradle Runner‘s GradleDaemon process and found that Metaspace memory growth was staggeringly fast – it hit OOM by the third test case:

Metaspace OOM

Test Logs

To see more detailed build output, I enabled logging for the Test task:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
task integrationTest(type: Test) {
description = 'Runs the integration tests.'
group = 'verification'
testClassesDirs = sourceSets.integrationTest.output.classesDirs
classpath = sourceSets.integrationTest.runtimeClasspath
mustRunAfter test
testLogging {
events "passed", "skipped", "failed", "standardOut", "standardError"
}
}

task functionalTest(type: Test) {
description = 'Runs the functional tests.'
group = 'verification'
testClassesDirs = sourceSets.functionalTest.output.classesDirs
classpath = sourceSets.functionalTest.runtimeClasspath
mustRunAfter test, integrationTest
testLogging {
events "passed", "skipped", "failed", "standardOut", "standardError"
}
}

This way, the detailed build process of the test project becomes visible:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
com.didiglobal.booster.android.gradle.v3_5.V35AppIntegrationTest > test AGPInterface#mergedManifests STANDARD_OUT
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:processResources
> Task :buildSrc:classes
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build
> Configure project :
file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/kotlin/integrationTest/
WARNING: The specified Android SDK Build Tools version (26.0.3) is ignored, as it is below the minimum supported version (28.0.3) for Android Gradle Plugin 3.5.0.
Android SDK Build Tools 28.0.3 will be used.
To suppress this warning, remove "buildToolsVersion '26.0.3'" from your build.gradle file, as each version of the Android Gradle Plugin now has a default version of the build tools.

> Task :preBuild UP-TO-DATE
> Task :preCnDebugBuild UP-TO-DATE
> Task :compileCnDebugAidl NO-SOURCE
> Task :compileCnDebugRenderscript NO-SOURCE
> Task :checkCnDebugManifest
> Task :generateCnDebugBuildConfig FROM-CACHE
> Task :javaPreCompileCnDebug FROM-CACHE
> Task :mainApkListPersistenceCnDebug
> Task :generateCnDebugResValues FROM-CACHE
> Task :generateCnDebugResources UP-TO-DATE
> Task :mergeCnDebugResources FROM-CACHE
> Task :createCnDebugCompatibleScreenManifests
> Task :processCnDebugManifest
> Task :processCnDebugResources
> Task :compileCnDebugJavaWithJavac FROM-CACHE
> Task :compileCnDebugSources UP-TO-DATE
> Task :mergeCnDebugShaders FROM-CACHE
> Task :compileCnDebugShaders FROM-CACHE
> Task :generateCnDebugAssets UP-TO-DATE
> Task :mergeCnDebugAssets FROM-CACHE
> Task :processCnDebugJavaRes NO-SOURCE
> Task :mergeCnDebugJavaResource FROM-CACHE
> Task :checkCnDebugDuplicateClasses FROM-CACHE
> Task :transformClassesWithDexBuilderForCnDebug
> Task :mergeExtDexCnDebug FROM-CACHE
> Task :mergeDexCnDebug FROM-CACHE
> Task :validateSigningCnDebug FROM-CACHE
> Task :signingConfigWriterCnDebug FROM-CACHE
> Task :mergeCnDebugJniLibFolders FROM-CACHE
> Task :mergeCnDebugNativeLibs FROM-CACHE
> Task :stripCnDebugDebugSymbols FROM-CACHE
> Task :packageCnDebug
> Task :transformClassesWithBoosterForCnDebug
> Task :assembleCnDebug
> Task :preCnReleaseBuild UP-TO-DATE
> Task :compileCnReleaseAidl NO-SOURCE
> Task :compileCnReleaseRenderscript NO-SOURCE
> Task :checkCnReleaseManifest
> Task :generateCnReleaseBuildConfig FROM-CACHE
> Task :javaPreCompileCnRelease FROM-CACHE
> Task :mainApkListPersistenceCnRelease
> Task :generateCnReleaseResValues FROM-CACHE
> Task :generateCnReleaseResources UP-TO-DATE
> Task :mergeCnReleaseResources FROM-CACHE
> Task :createCnReleaseCompatibleScreenManifests
> Task :processCnReleaseManifest
> Task :processCnReleaseResources
> Task :compileCnReleaseJavaWithJavac FROM-CACHE
> Task :compileCnReleaseSources UP-TO-DATE
> Task :prepareLintJar UP-TO-DATE
> Task :lintVitalCnRelease FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':lintVitalCnRelease'.
> java.lang.OutOfMemoryError: Metaspace
at com.android.tools.lint.checks.BuiltinIssueRegistry.<clinit>(BuiltinIssueRegistry.java:363)
at com.android.tools.lint.gradle.LintGradleExecution.createIssueRegistry(LintGradleExecution.java:375)
at com.android.tools.lint.gradle.LintGradleExecution.runLint(LintGradleExecution.java:216)
at com.android.tools.lint.gradle.LintGradleExecution.lintSingleVariant(LintGradleExecution.java:385)
at com.android.tools.lint.gradle.LintGradleExecution.analyze(LintGradleExecution.java:91)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.android.tools.lint.gradle.api.ReflectiveLintRunner.runLint(ReflectiveLintRunner.kt:38)
at com.android.build.gradle.tasks.LintBaseTask.runLint(LintBaseTask.java:100)
at com.android.build.gradle.tasks.LintPerVariantTask.lint(LintPerVariantTask.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:103)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:48)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)


* Try:

Run with
--info
or
--debug
option to get more log output. Run with
--scan
to get full insights.

From the stack trace above, the culprit was the lintVitalCnRelease task. So I tried disabling lintVitalCnRelease:

1
2
3
4
5
6
7
8
android {

...

lintOptions {
checkReleaseBuilds false
}
}

After running the test cases again and analyzing with VisualVM, the Metaspace growth curve looked like this:

Normal Metaspace

Of course, aside from disabling lintVital, you can also increase the Metaspace size via gradle.properties:

1
org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -Xms512m -Xmx2048m