We’re pleased to announce an important interoperability fix for wolfSSH and WinSCP, introduced in Pull Request #832. This update resolves connection issues when using WinSCP with the SCP protocol in Default shell mode, enabling wolfSSH to work seamlessly with one of the most popular SFTP and SCP clients for Windows.
Download wolfSSL →
About WinSCP
WinSCP is one of the most widely used SFTP and SCP clients for Windows, used by millions of users worldwide for secure file transfers. When integrating wolfSSH with WinSCP, we discovered that WinSCP was failing to connect when configured to use the SCP protocol with the Default shell option. After the SSH handshake completed successfully, WinSCP would hang indefinitely, unable to proceed with file transfer operations.
The Problem and Solution
The root cause was a fundamental difference in how WinSCP and wolfSSH handled shell sessions. WinSCP opens a shell subsystem without requesting a pseudo-terminal (pty) and expects the server to create a non-interactive shell. However, wolfSSH was treating all shell sessions as interactive by default, sending terminal control characters and prompts that confused WinSCP’s SCP implementation.
Understanding PTY Requests: In the SSH protocol, a pty-req message (defined in RFC 4254 section 6.2) signals that the client expects interactive terminal behavior with terminal control characters, line editing, and interactive prompts. When a client opens a shell subsystem without sending a pty-req message, it typically expects a non-interactive shell suitable for automated command execution or protocol implementations like SCP.
The Fix: The solution involves tracking whether a pty-req message has been received from the client and adjusting shell behavior accordingly. If no pty-req is received, wolfSSH now creates a non-interactive shell with separate pipes for stdin, stdout, and stderr, similar to how forced commands are handled.
Implementation Details
The fix was implemented through a series of commits that refined the approach:
Initial Implementation: Added a ptyReq flag to track whether a pty-req message was received, along with a public API function wolfSSH_ReceivedPtyReq() to check the status. The shell subsystem in wolfsshd was modified to check the pty-req status and create appropriate pipes for non-interactive shells.
Channel-Level Refactoring: Based on code review feedback, the implementation was refactored to move the pty flag from the session level to the channel level, which is more architecturally correct since SSH channels are independent and each can have its own pty status.
API Simplification: The final commit simplified the API to wolfSSH_ChannelIsPty(const WOLFSSH_CHANNEL* channel), providing a cleaner interface for checking if a specific channel has pty capabilities.
Impact and Benefits
This fix improves wolfSSH’s interoperability with WinSCP and potentially other SSH clients that use similar patterns for non-interactive shell sessions. After implementing this fix, WinSCP can now successfully connect to wolfSSH servers using the SCP protocol with Default shell mode. The implementation correctly distinguishes between interactive and non-interactive shell sessions as intended by the SSH protocol, while maintaining functionality for existing interactive shell sessions and forced commands.
For organizations using WinSCP for Windows-based file transfers, this fix enables seamless integration with wolfSSH servers. It also benefits any automation tools or clients that open shell channels without requesting pseudo-terminals.
If you have questions about wolfSSH interoperability with other SSH clients or specific integration needs, please contact us at facts@wolfSSL.com or call us at +1 425 245 8247.
Download wolfSSL Now