[build-tools] support Gradle build cache for Android builds#3510
[build-tools] support Gradle build cache for Android builds#3510AbbanMustafa wants to merge 14 commits intomainfrom
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3510 +/- ##
==========================================
- Coverage 53.62% 53.58% -0.03%
==========================================
Files 815 816 +1
Lines 34622 34752 +130
Branches 7192 7223 +31
==========================================
+ Hits 18562 18618 +56
- Misses 15979 16053 +74
Partials 81 81 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Subscribed to pull request
Generated by CodeMention |
|
|
||
| const GRADLE_CACHES_DIR = '.gradle/caches'; | ||
|
|
||
| export function createRestoreGradleCacheFunction(): BuildFunction { |
There was a problem hiding this comment.
I don't see reason to add a new function? What is your reasoning re: having a single vs having multiple cache functions?
We talked before about introducing config like
cache: true
cache:
node_modules: true
xcode: false
gradle: falseor sth like that to the workflow YAML. If we had cache config in one object I don't think it makes sense to require user to set up different functions in custom jobs?
Shouldn't this be part of {restore,save}_build_cache?
There was a problem hiding this comment.
I don't feel strongly about separate functions, can definitely fold this into {restore,save}_build_cache. I mainly wanted this PR to land the functionality so we can get feedback on how this works in prod, and then follow up with this and Xcode caches to set up introducing the proper config as a next step.
| logger.info(`Running 'gradlew ${gradleCommand}' in ${androidDir}`); | ||
| await fs.chmod(path.join(androidDir, 'gradlew'), 0o755); | ||
| const verboseFlag = ctx.env['EAS_VERBOSE'] === '1' ? '--info' : ''; | ||
| const buildCacheFlag = ctx.env['GRADLE_CACHE'] === '1' ? '--build-cache' : ''; |
There was a problem hiding this comment.
So sad Gradle can't be ordered to use build cache with an environment variable…
There was a problem hiding this comment.
ah yes we can enable with gradle.properties
There was a problem hiding this comment.
Oh I didn't mean gradle.properties would necessarily be better. I like the explicitness of --build-cache next to ./gradlew. Or maybe it would? Can we put gradle.properties anywhere, or only in the project?
\n... be sufficient?
There was a problem hiding this comment.
yes we can do so via the project properties. I write to the file if we find the property has not been set daa3873#diff-db8c2375feedbfe1a1b25cbc4ef8375f0d592509438f4babcc5db95d6f28cdc1R57-R69
| import { generateGradleCacheKeyAsync } from '../../utils/gradleCacheKey'; | ||
|
|
||
| const GRADLE_CACHES_DIR = '.gradle/caches'; | ||
| const CACHE_SUBDIRS = ['build-cache-1', 'modules-2']; |
There was a problem hiding this comment.
I think we shouldn't cache modules-2. As you mentioned, those are Maven/Gradle artifacts and dependencies. To help with that we have the Maven cache proxy. If we start caching these for all builds we'll go from storing one artifact in the Maven cache for all the builds to use to storing one artifact for every build separately. This is going to inflate network bandwidth and cache size a lot.
Can you confirm if caching just the build-cache-1 helps too? https://docs.gradle.org/current/userguide/build_cache.html does not mention modules-2, it does however talk about build-cache.
There was a problem hiding this comment.
I saw speed benefits when restoring modules-2, but in terms of the gradle's reported cache hit, it performs the same. So yes I was just observing locally the benefits of what our maven repo provides for the worker. So I will remove modules-2
| const expoApiServerURL = nullthrows(env.__API_SERVER_URL, '__API_SERVER_URL is not set'); | ||
|
|
||
| logger.info(`Compressing Gradle caches (${existingDirs.join(', ')})...`); | ||
| const { archivePath } = await compressCacheAsync({ |
There was a problem hiding this comment.
What about configuring cache pruning?
| workingDirectory: string; | ||
| env: Record<string, string | undefined>; | ||
| secrets?: { robotAccessToken?: string }; | ||
| }): Promise<boolean> { |
There was a problem hiding this comment.
| }): Promise<boolean> { | |
| }): Promise<{ restored: boolean }> { |
or cacheHit or sth like that?
| const properties = AndroidConfig.Properties.parsePropertiesFile(gradlePropertiesContent); | ||
|
|
||
| let modified = false; | ||
| if (!properties.some(p => p.type === 'property' && p.key === 'org.gradle.caching')) { | ||
| properties.push({ type: 'property', key: 'org.gradle.caching', value: 'true' }); | ||
| modified = true; | ||
| } | ||
| if (!properties.some(p => p.type === 'property' && p.key === 'org.gradle.cache.cleanup')) { | ||
| properties.push({ type: 'property', key: 'org.gradle.cache.cleanup', value: 'ALWAYS' }); | ||
| modified = true; | ||
| } | ||
| if (modified) { | ||
| await fs.promises.writeFile( | ||
| gradlePropertiesPath, | ||
| AndroidConfig.Properties.propertiesListToString(properties) | ||
| ); | ||
| } |
There was a problem hiding this comment.
Should we allow custom properties for org.gradle.caching and org.gradle.cache.cleanup if a user uses EAS_GRADLE_CACHE? To put it differently, shouldn't EAS_GRADLE_CACHE override whatever the user has set?
There was a problem hiding this comment.
Yes, I agree if the user opts in then that should set precedent for the properties. Can also tweak more how the customization looks once this our
| const cacheKey = await generateGradleCacheKeyAsync(workingDirectory); | ||
| logger.info(`Restoring Gradle cache key: ${cacheKey}`); | ||
|
|
||
| const jobId = nullthrows(env.EAS_BUILD_ID, 'EAS_BUILD_ID is not set'); |
There was a problem hiding this comment.
Why do we do nullthrows sometimes inside try and sometimes outside try?
| }); | ||
|
|
||
| await fs.promises.mkdir(gradleCachesPath, { recursive: true }); | ||
| await spawn('tar', ['xzf', archivePath, '-C', gradleCachesPath], { logger }); |
There was a problem hiding this comment.
Why can't we use the regular cache utilities?
| } | ||
|
|
||
| if (cacheHit) { | ||
| logger.info('Gradle cache was restored — skipping save'); |
There was a problem hiding this comment.
Why are we skipping save if restored?
| return; | ||
| } | ||
|
|
||
| const existingDirs = []; |
| paths: ['gradle-caches'], | ||
| key: cacheKey, | ||
| keyPrefixes: [GRADLE_CACHE_KEY_PREFIX], |
There was a problem hiding this comment.
This does not match compressCacheAsync in saveBuildCache which means version won't match and cache will never get restored? Did you test this?
There was a problem hiding this comment.
Its a bit confusing with the parameter naming, but paths in compressCacheAsync and paths in uploadCacheAsync/downloadCacheAsync are for different purpose. And we just need it consistent between the up/download functions. I believe you meant upload since that is what determines the version, which was the same https://github.com/expo/eas-cli/pull/3510/changes#diff-b282ed96909cf3405e277859d596a3617f65db34f4f22319efcd062a549a3430R207
I've aligned them all now to just have ['build-cache-1'] as path to be consistent with how ccache does it and avoid confusion.
|
⏩ The changelog entry check has been skipped since the "no changelog" label is present. |
sjchmiela
left a comment
There was a problem hiding this comment.
Much, much nicer! Left a few questions
|
|
||
| const propsToSet: Record<string, string> = { | ||
| 'org.gradle.caching': 'true', | ||
| 'org.gradle.cache.cleanup': 'ALWAYS', |
There was a problem hiding this comment.
Looks like we'll need to either handle Gradle 9.0 in a dedicated way or move to init script as per https://docs.gradle.org/current/userguide/upgrading_major_version_9.html#removal_of_org_gradle_cache_cleanup. We want to support Gradle 9 https://exponent-internal.slack.com/archives/C1QP38NQ5/p1772688272258629
There was a problem hiding this comment.
Also, I think the default expiration times are too high for us?
| const properties = AndroidConfig.Properties.parsePropertiesFile(gradlePropertiesContent); | ||
|
|
||
| const propsToSet: Record<string, string> = { | ||
| 'org.gradle.caching': 'true', | ||
| 'org.gradle.cache.cleanup': 'ALWAYS', | ||
| }; | ||
|
|
||
| for (const [key, value] of Object.entries(propsToSet)) { | ||
| const existing = properties.find(p => p.type === 'property' && p.key === key); | ||
| if (existing && existing.type === 'property') { | ||
| existing.value = value; | ||
| } else { | ||
| properties.push({ type: 'property', key, value }); | ||
| } | ||
| } | ||
|
|
||
| await fs.promises.writeFile( | ||
| gradlePropertiesPath, | ||
| AndroidConfig.Properties.propertiesListToString(properties) | ||
| ); |
There was a problem hiding this comment.
Is this better than ⬇️ ? (sth like)
| const properties = AndroidConfig.Properties.parsePropertiesFile(gradlePropertiesContent); | |
| const propsToSet: Record<string, string> = { | |
| 'org.gradle.caching': 'true', | |
| 'org.gradle.cache.cleanup': 'ALWAYS', | |
| }; | |
| for (const [key, value] of Object.entries(propsToSet)) { | |
| const existing = properties.find(p => p.type === 'property' && p.key === key); | |
| if (existing && existing.type === 'property') { | |
| existing.value = value; | |
| } else { | |
| properties.push({ type: 'property', key, value }); | |
| } | |
| } | |
| await fs.promises.writeFile( | |
| gradlePropertiesPath, | |
| AndroidConfig.Properties.propertiesListToString(properties) | |
| ); | |
| await fs.promises.writeFile( | |
| gradlePropertiesPath, | |
| `${gradlePropertiesContent}\n\norg.gradle.caching=true\norg.gradle.cache.cleanup=ALWAYS\n` | |
| ); |
?
| jobId, | ||
| expoApiServerURL, | ||
| robotAccessToken, | ||
| paths: ['build-cache-1'], |
There was a problem hiding this comment.
Why not use path.join(gradleCachesPath, 'build-cache-1')?



Why
Android builds currently recompile all Kotlin/Java code from scratch on every build. We already persist ccache between builds to speed up native C/C++ compilation (NDK), but ccache doesn't help with the Gradle/Kotlin/Java layer — which is where the majority of Android build time is spent. This adds Gradle build cache support to close that gap, leading to more optimal Android builds.
How
All behavior is gated behind
GRADLE_CACHE=1— when unset, everything is a no-op.Cache key is android-gradle-cache-{lockfile_hash}, derived from the project's package manager lockfile. Changing dependencies invalidates the cache; code-only changes reuse it. On miss, falls back to prefix match
(android-gradle-cache-) to find the closest available cache.
Two new build phases are added to the Android builder:
Restore downloads and extracts the cache archive into
~/.gradle/caches/. Save compresses and uploads two specific subdirectories from~/.gradle/caches/:We selectively cache
build-cache-1rather than the full ~/.gradle/caches/ directory. This subdirectory covers the highest-impact layers: compiled task outputs (classes, dex, resources). The remaining contents — transforms, journal, configuration-cache, etc. — are either derived from modules-2 (transforms are regenerated from cached dependencies onfirst build) or are Gradle bookkeeping with negligible build time impact.
Test Plan
Validated locally, observed build time improvement of 4min -> 40sec