Recently I was preparing tests before releasing Boosterv3.0.0. To ensure Booster‘s quality, I adapted it to every version of the Android Gradle Plugin from 3.0.0 to 4.1.0 and wrote a large number of integration tests. While running the test cases, the debug build tests went smoothly at first. Then on a whim I decided to add release builds too. To my surprise, the test suite got stuck on the Android Gradle Plugin3.5.0 case and wouldn’t budge. After many retries with the same result, I checked and found it was already OOM – a pile of hprof files had been dumped. Looking at the JUnit test report, I discovered it was a Metaspace overflow.
> Configure project : file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/java/main/ file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/kotlin/main/ file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/tmp/kapt3/classes/main/ file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/resources/main file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-compat/build/libs/booster-android-gradle-compat-3.0.0-SNAPSHOT.jar file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.3.61/2e07c9a84c9e118efb70eede7e579fd663932122/kotlin-reflect-1.3.61.jar file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.3.61/4702105e97f7396ae41b113fdbdc180ec1eb1e36/kotlin-stdlib-1.3.61.jar file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.3.61/65abb71d5afb850b68be03987b08e2c864ca3110/kotlin-stdlib-common-1.3.61.jar file:/Users/johnsonlee/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/kotlin/integrationTest/ WARNING: The specified Android SDK Build Tools version (26.0.3) is ignored, as it is below the minimum supported version (28.0.3) for Android Gradle Plugin 3.5.0. Android SDK Build Tools 28.0.3 will be used. To suppress this warning, remove "buildToolsVersion '26.0.3'" from your build.gradle file, as each version of the Android Gradle Plugin now has a default version of the build tools.
* Try: Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is: java.lang.OutOfMemoryError: Metaspace
* Get more help at https://help.gradle.org
BUILD FAILED in 24s
at org.gradle.testkit.runner.internal.DefaultGradleRunner$2.execute(DefaultGradleRunner.java:255) at org.gradle.testkit.runner.internal.DefaultGradleRunner$2.execute(DefaultGradleRunner.java:251) at org.gradle.testkit.runner.internal.DefaultGradleRunner.run(DefaultGradleRunner.java:324) at org.gradle.testkit.runner.internal.DefaultGradleRunner.build(DefaultGradleRunner.java:251) at io.bootstage.testkit.gradle.rules.GradleExecutor.finished(GradleExecutor.kt:50) at org.junit.rules.TestWatcher.finishedQuietly(TestWatcher.java:117) at org.junit.rules.TestWatcher.access$400(TestWatcher.java:46) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:64) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164) at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) at java.lang.Thread.run(Thread.java:748)
Gradle Testing
Running Gradle plugin tests requires Gradle TestKit. Before each @Test method, a temporary Android Gradle project is dynamically created, and after the @Test method, the Gradle Runner executes the assemble task. Within the @Test method body, you can manipulate this dynamically generated Android Gradle project. For example:
When the Gradle Runner executes a test build task, it spawns a new GradleDaemon process. By default, the Gradle Runner creates a directory similar to ~/.gradle under $TMPDIR for caching files. When the OOM first appeared, I suspected it was related to the Gradle Runner, but after resetting the Gradle Runner‘s directory, the OOM persisted.
Profiling
Since the OOM occurred in Metaspace, it was most likely related to class loading. So I used VisualVM to profile the Gradle Runner‘s GradleDaemon process and found that Metaspace memory growth was staggeringly fast – it hit OOM by the third test case:
Test Logs
To see more detailed build output, I enabled logging for the Test task:
com.didiglobal.booster.android.gradle.v3_5.V35AppIntegrationTest > test AGPInterface#mergedManifests STANDARD_OUT > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy NO-SOURCE > Task :buildSrc:processResources > Task :buildSrc:classes > Task :buildSrc:jar > Task :buildSrc:assemble > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build > Configure project : file:/Users/johnsonlee/Workspace/github/didi/booster/booster-android-gradle-v3_5/build/classes/kotlin/integrationTest/ WARNING: The specified Android SDK Build Tools version (26.0.3) is ignored, as it is below the minimum supported version (28.0.3) for Android Gradle Plugin 3.5.0. Android SDK Build Tools 28.0.3 will be used. To suppress this warning, remove "buildToolsVersion '26.0.3'" from your build.gradle file, as each version of the Android Gradle Plugin now has a default version of the build tools.
* What went wrong: Execution failed for task ':lintVitalCnRelease'. > java.lang.OutOfMemoryError: Metaspace at com.android.tools.lint.checks.BuiltinIssueRegistry.<clinit>(BuiltinIssueRegistry.java:363) at com.android.tools.lint.gradle.LintGradleExecution.createIssueRegistry(LintGradleExecution.java:375) at com.android.tools.lint.gradle.LintGradleExecution.runLint(LintGradleExecution.java:216) at com.android.tools.lint.gradle.LintGradleExecution.lintSingleVariant(LintGradleExecution.java:385) at com.android.tools.lint.gradle.LintGradleExecution.analyze(LintGradleExecution.java:91) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.android.tools.lint.gradle.api.ReflectiveLintRunner.runLint(ReflectiveLintRunner.kt:38) at com.android.build.gradle.tasks.LintBaseTask.runLint(LintBaseTask.java:100) at com.android.build.gradle.tasks.LintPerVariantTask.lint(LintPerVariantTask.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:103) at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:48) at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:41) at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:28)
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
From the stack trace above, the culprit was the lintVitalCnRelease task. So I tried disabling lintVitalCnRelease:
1 2 3 4 5 6 7 8
android {
...
lintOptions { checkReleaseBuilds false } }
After running the test cases again and analyzing with VisualVM, the Metaspace growth curve looked like this:
Of course, aside from disabling lintVital, you can also increase the Metaspace size via gradle.properties: