[SFTP] Race Condition during Rekeying: "Message type 93 is not valid" or Connection Drop #1764
Description
Description
We are encountering critical connection failures during SFTP uploads under high network throughput. The issue appears to be a race condition during the SSH Rekeying phase.
When the server triggers a Key Rotation (due to RekeyLimit), the client fails to pause the data stream immediately. Consequently, the client sends a SSH_MSG_CHANNEL_DATA packet while the server has entered the Key Exchange state and expects SSH_MSG_KEXINIT, leading to a protocol violation.
Exceptions & Stack Traces
We observe two different exceptions depending on the server software/configuration:
Variant 1: Protocol Violation (Server sends Disconnect message)
Renci.SshNet.Common.SshConnectionException: Message type 93 is not valid.
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle)
at Renci.SshNet.Channels.Channel.SendData(Byte[] data)
at Renci.SshNet.Channels.Channel.SendMessage(Message message)
at Renci.SshNet.Channels.Channel.SendData(Byte[] buffer, Int32 offset, Int32 size)
at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, Action`1 writeStatusCallback)
Variant 2: Connection Drop (Server resets TCP connection)
Renci.SshNet.Common.SshConnectionException: Client not connected.
at Renci.SshNet.Common.AsyncResult.EndInvoke()
at Renci.SshNet.SftpClient.EndUploadFile(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at FtpUtils.FtpUtils.SftpTransferFileAsync(...)
Steps to Reproduce
Configure an SFTP server with a low RekeyLimit (e.g., RekeyLimit 500M 0).
Establish a high-speed connection (e.g., > 500 Mbps).
Upload a file significantly larger than the limit.
Result: Connection crashes exactly when the limit is reached.
Analysis
The library seems unable to handle the state transition to KeyExchange fast enough when the outgoing socket buffer is saturated with file data. Confirmed Workaround: The issue is resolved only by setting RekeyLimit none on the server side, which confirms the issue is isolated to the rekeying phase.
Additional Info
- Reproducible on the latest revision 2025.1.0 as well as older versions.
- Reproducible with both synchronous UploadFile and asynchronous UploadFileAsync / BeginUploadFile implementations.
Question
Is there any client-side configuration or internal buffer setting that could mitigate this race condition without changing the server config? Or any other trick. Thanks!