Thanks for the tips. Comparing differences in debug output from Linux (where things work fine) and Pico helped me to track down the issue. It wasn't stack overflow, but too small network (rx) buffer that wasn't large enough to fit the ML-KEM key exchange.

I know that running SSH server (and everything else) in 264Kb of RAM is pushing the limits, but looks like its possible...

I enabled debugging (DEBUG_WOLFSSH), and from the logs it would seem that it is the server that gets stuck in a "loop", getting same error repeatedly:

accept error: SERVER_KEXINIT_SENT, -1010

Log can be found here: https://gist.github.com/tjko/53122b4165 … 4ec6f0c5f6

I'm seeing this on Raspberry Pi Pico W (RP2040).

Here is link to changes I made to enable ML-KEM support: https://github.com/tjko/fanpico/compare/mlkem

If you have a Pico W (or Pico 2 W should yield same results), you should be able to compile firmware image from "mlkem" branch to reproduce the issue:
https://github.com/tjko/fanpico/tree/mlkem


It would seem like its the server that fails to respond to SSH packet "30" (?), since with other kex algorihms client gets response:

debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes128-gcm@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
...

(and SSH connection negotiation succeeds normally)

As new OpenSSH versions have started nagging about not using post-quantum safe key exchange algorithms, I decided to test enabling ML-KEM on wolfSSH.

I added following in user_settings.h:

#define WOLFSSL_HAVE_MLKEM
#define WOLFSSL_WC_MLKEM
#define WOLFSSL_SHAKE128
#define WOLFSSL_SHAKE256

And that seemed to work, as (after re-compiling) wolfSSH server started advertising two new "mlkem" algorithms:

debug2: peer server KEXINIT proposal
debug2: KEX algorithms: mlkem768x25519-sha256,mlkem768nistp256-sha256,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@openssh.com,aes192-gcm@openssh.com,aes128-gcm@openssh.com
debug2: ciphers stoc: aes256-gcm@openssh.com,aes192-gcm@openssh.com,aes128-gcm@openssh.com
debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none
debug2: compression stoc: none

However, when trying to connect using OpenSSH client (OpenSSH_10.2p1, LibreSSL 3.3.6), connection seems to hang indefinitely in key exchange:

debug1: kex: algorithm: mlkem768x25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes128-gcm@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

(client hangs here...)


If I try any of the other advertised kex algorithms (curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256) connection works just fine.... (I was testing on latest 'version' pulled from github)

I would think userAuthResultCb would be much more useful if it would always get called at the end of client authentication process. That would make it much more easier to track if client succeeded in authentication or not...

I added "SSH server" that uses wolfSSH in my personal (OSWH) project: https://github.com/tjko/fanpico
It seems to work surprisingly well on RP2040 (Pico W), when using ECC based authentication...

When experimenting with wolfSSH on RP2040 MCU, I noticed couple things.

It seems "UserAuthResult" callback (set using wolfSSH_SetUserAuthResult()) doesn't always get called. It gets called when client is authenticating using "publickey" authentication, but seems like it does not get called at all when "password" authentication is used by client.  This seems like a bug? (Or is there another way to easily track successful/failed logins?)

Looking src/internal.c, it seems callback is only called from DoUserAuthRequestPublicKey() function, so that seems to explain why it does not get called at all, if other type authentication is being used....


Another "odd" thing I noticed is with the "UserAuth" callback (set using wolfSSH_SetUserAuth()). I see it getting called twice when public key authentication succeeds...

Example (debug logs from server side):

user_auth_cb(8,20041918,20024E10): 2
user_auth_cb(2,20041AA0,20024E10): 6
user_auth_cb(2,20041870,20024E10): 0
user_auth_cb(2,20041AB8,20024E10): 0
user_auth_result_cb(0,20041AB8,20024E10): OK

Client log (OpenSSH):

debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-nistp256>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/test/.ssh/id_rsa
debug1: Offering public key: /home/test/.ssh/id_ecdsa ECDSA SHA256:lHQBcPRpky2hfMiffYnoVfCww9XL/XC0JZ8CGaPLYCM
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/test/.ssh/id_ecdsa_sk
debug1: Offering public key: /home/test/.ssh/id_ed25519 ED25519 SHA256:NZHx/BI7BCGUS2WVIG+qdrwd/YaH8xk8GgN19L/QJdg
debug1: Server accepts key: /home/test/.ssh/id_ed25519 ED25519 SHA256:NZHx/BI7BCGUS2WVIG+qdrwd/YaH8xk8GgN19L/QJdg
Authenticated to 192.168.4.17 ([192.168.4.17]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: channel 0: setting env LC_ALL = "en_US.UTF-8"

However, when client uses password authentication there is no "second" call to callback function after successful authentication...

Server side:

user_auth_cb(8,20041870,20024E10): 2
user_auth_cb(2,20041AA8,20024E10): 6
user_auth_cb(2,20041858,20024E10): 6
user_auth_cb(1,20041AC0,20024E10): 0

(notice, the "UserAuthResult" callback did not get called at all...)

Client side:

debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/test/.ssh/id_rsa
debug1: Offering public key: /home/test/.ssh/id_ecdsa ECDSA SHA256:lHQBcPRpky2hfMiffYnoVfCww9XL/XC0JZ8CGaPLYCM
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/test/.ssh/id_ecdsa_sk
debug1: Offering public key: /home/test/.ssh/id_ed25519 ED25519 SHA256:NZHx/BI7BCGUS2WVIG+qdrwd/YaH8xk8GgN19L/QJdg
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/test/.ssh/id_ed25519_sk
debug1: Trying private key: /home/test/.ssh/id_xmss
debug1: Trying private key: /home/test/.ssh/id_dsa
debug1: Next authentication method: password
admin@192.168.4.17's password: 
Authenticated to 192.168.4.17 ([192.168.4.17]:22) using "password".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: channel 0: setting env LC_ALL = "en_US.UTF-8"