Hi Yin,
It is possible you are hitting a heap / stack limit or there is some other configuration issue. I highly recommend looking at the user_settings.h file in our examples repository here: https://github.com/wolfSSL/wolfssl-exam … r/RPi-Pico
Options such as WOLFSSL_SMALL_STACK is highly recommended. WOLFSSL_SP_SMALL and RSA_LOW_MEM can help, even though they are not used in the examples.
Also, if you are using ARM assembly options, make sure you compile with `WOLFSSL_SP_ARM_CORTEX_M_ASM` defined, you can see this in the CMakeList.txt for the examples.
If you don't have one already, I highly recommend obtaining an SWD debug probe for the Pico, or using Raspberry Pi's examples on converting a second Pico into an SWD debug probe. This way you will be able to observe what the problem might be.
Whilst we haven't tested ML_DSA with the RP2040 or RP2350, it works on similar ARM platforms, so should work without issue.
Kind Regards
Andrew Hutchings, wolfSSL