Repository navigation
feat(client): add BatchExecutor and batch.threads option - #460
dheerajsngh wants to merge 4 commits into
Conversation
Adds a package-private BatchExecutor modeled on the hadoop-connectors implementation, plus only the changes analytics-core needs: a reusable awaitRequestsCompletion() phase barrier that drains every request before throwing, bounded memory by dropping succeeded futures, onSuccess isolated from onFailure, InterruptedIOException on owner interruption, and a shutdown() that always stops the pool. Documents a single-owner threading contract. Adds GcsClientOptions batch.threads (default 15, range 0..1024; defaults mirror hadoop-connectors pending benchmarking).
Cancels the never-started requests that shutdownNow() returns so their futures always complete and a later awaitRequestsCompletion()/shutdown() cannot block. Documents that a phase must be fully queued before awaitRequestsCompletion() is called, and adds a test for reporting the same exception instance from several requests without self-suppression.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new BatchExecutor component designed to handle concurrent GCS RPC operations efficiently. By providing a bounded, transport-agnostic executor, it enables future directory metadata operations to perform concurrent requests without risking memory exhaustion or unbounded future retention. The change also includes necessary configuration options to tune this concurrency, ensuring robust performance and resource management. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #460 +/- ##
==========================================
- Coverage 98.47% 98.25% -0.22%
==========================================
Files 79 80 +1
Lines 3922 4064 +142
Branches 457 483 +26
==========================================
+ Hits 3862 3993 +131
- Misses 32 40 +8
- Partials 28 31 +3
🚀 New features to boost your workflow:
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces BatchExecutor to run independent GCS requests on a bounded thread pool, along with corresponding unit tests and configuration options in GcsClientOptions to control the thread count. The review feedback suggests adhering to the repository style guide by avoiding the static import of Guava's newDirectExecutorService. Additionally, it recommends improving error handling in BatchExecutor.execute to ensure fatal JVM Errors are prioritized and not suppressed if a callback's onFailure method throws an exception.
📊 Hermetic JMH Benchmark Summary (PR vs Main)PR: Published Assets: 🌐 Interactive HTML Report | 📄 Markdown Report | 📦 Raw JMH JSON | 🔬 Kokoro / Sponge Artifacts Suite-Level Executive Summary
📋 Full Hermetic Benchmark Breakdown (45 cases)
|
Type of Change
feat: A new featurefix: A bug fixdocs: Documentation only changesstyle: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)refactor: A code change that neither fixes a bug nor adds a featureperf: A code change that improves performancetest: Adding missing tests or correcting existing testschore: Changes to the build process or auxiliary tools and libraries such as documentation generationDescription
What?
BatchExecutorincom.google.cloud.gcs.analyticscore.clientto run independent GCS requests concurrently on a bounded daemonThreadPoolExecutor(numThreadscore/max threads, 10s keep-alive,numThreads * 20bounded queue with caller-runs backpressure while active, orMoreExecutors.newDirectExecutorService()whennumThreads == 0):queue(Callable<T>, FutureCallback<? super T>): dispatches per-request results/errors to an optional callback withonSuccessisolated outside thetask.call()try/catchblock, rethrowsErrors, and periodically prunes completed futures so retained memory stays bounded across large streaming listings.awaitRequestsCompletion(): reusable phase barrier under a documented single-owner threading contract (with callback retry re-queuing supported) that awaits all queued requests before throwing the firstIOException(attaching up to 100 suppressed failures), or cancels remaining requests and restores interrupt status when the waiting thread is interrupted.shutdown()andisIdle(): drains queued work and always terminates the pool infinally(1s grace period beforeshutdownNow(), cancelling any never-started queued tasks so futures always complete).batch.threads(getBatchThreads()/Builder.setBatchThreads(int)) toGcsClientOptionswith default15and validation bounds0..1024.BatchExecutorTestandGcsClientOptionsTestcovering pool and direct execution, backpressure, callback isolation, phase barriers, callback retries, bounded future retention, interruption, forced shutdown, and option parsing.Why?
Futureinstances in memory.google-cloud-storagesupports single-objectdeleteandmoveBlobuniformly across both HTTP and gRPC transports, whereas JSON multipart batching (Storage.batch()) is unsupported on gRPC and executes sub-request chunks sequentially; a bounded client-side executor provides transport-agnostic concurrency with caller-runs backpressure.Checklist
feat(core): ...)Generated/Assisted by Agent? Yes