-
Notifications
You must be signed in to change notification settings - Fork 621
Implement graceful shutdown for Garnet server #1448
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Implement graceful shutdown for Garnet server #1448
Conversation
Adds a graceful shutdown mechanism to the Garnet server, ensuring new connections are stopped, active connections are awaited, and data is safely persisted (AOF commit and checkpoint) before exit. Updates include new ShutdownAsync logic in GarnetServer, StopListening support in server classes, and integration of shutdown handling in both Windows service and console entry points.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request implements a comprehensive graceful shutdown mechanism for the Garnet server, addressing GitHub issues #1382 and #1390. The implementation provides a coordinated shutdown process that stops accepting new connections, waits for active connections to complete, commits the Append-Only File (AOF), and takes checkpoints for tiered storage—all within configurable timeouts. This enhancement significantly improves data durability and operational reliability during server restarts and deployments.
Key Changes:
- Added
ShutdownAsyncmethod to coordinate graceful shutdown with connection draining, AOF commit, and checkpointing - Introduced
StopListeningmethod to cleanly stop accepting new connections while preserving existing ones - Updated both console application and Windows service entry points to use the new graceful shutdown flow with proper timeout handling
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| main/GarnetServer/Program.cs | Converted Main to async, added shutdown signal handlers (Ctrl+C, ProcessExit), implemented graceful shutdown with timeout and disposal coordination |
| libs/host/GarnetServer.cs | Added ShutdownAsync method orchestrating stop-listening, connection waiting, AOF commit, and checkpoint operations |
| libs/server/Servers/IGarnetServer.cs | Extended interface with StopListening method for graceful shutdown support |
| libs/server/Servers/GarnetServerBase.cs | Added virtual StopListening method with default no-op implementation |
| libs/server/Servers/GarnetServerTcp.cs | Implemented StopListening by closing listen socket and tracking listening state |
| libs/server/Servers/StoreApi.cs | Added TakeCheckpoint method and storage tier/AOF enabled properties for shutdown logic |
| hosting/Windows/Garnet.worker/Worker.cs | Updated StopAsync to call ShutdownAsync before disposal, improved Dispose pattern with base.Dispose and GC.SuppressFinalize |
| hosting/Windows/Garnet.worker/Program.cs | Added HostOptions configuration for 5-second shutdown timeout and System namespace import |
Comments suppressed due to low confidence (1)
main/GarnetServer/Program.cs:16
- Disposable 'CancellationTokenSource' is created but not disposed.
var shutdownCts = new CancellationTokenSource();
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
This pull request introduces a comprehensive and robust graceful shutdown mechanism for the Garnet server, both for Windows service and console application scenarios. The changes ensure that when a shutdown is requested (via service stop, Ctrl+C, or process exit), the server stops accepting new connections, waits for existing connections to finish, commits all data (AOF), and takes a checkpoint if necessary, all within a configurable timeout. This significantly improves data durability and operational reliability during shutdowns.
Mainly goal is Close #1382 and Resolve #1390
This PR reflect #1383 (comment)
Graceful Shutdown Implementation
ShutdownAsyncmethod tomain/GarnetServer/GarnetServerthat orchestrates the graceful shutdown process: stops accepting new connections, waits for active connections to finish (with timeout), commits AOF, and takes a checkpoint if tiered storage is enabled.Worker.StopAsync) and console app (Program.Main) to use the newShutdownAsyncmethod, ensuring consistent and graceful shutdown behavior in both entrypoints. [1] [2]Server Interface and Networking Enhancements
libs/server/Servers/IGarnetServer) and base classes to support stopping listening for new connections via a newStopListeningmethod, and implemented this for TCP servers to close the listen socket cleanly. [1] [2] [3]Data Durability and Checkpointing
libs/server/Servers/StoreApito take a checkpoint and to check if AOF or storage tier is enabled, supporting the shutdown flow for data durability.Infrastructure and Code Quality Improvements
base.Dispose()and suppressing finalization.Configuration and Timeout Handling
These changes collectively make server shutdowns safer and more reliable, reducing the risk of data loss or corruption during restarts or deployments.