Hi Carlos,
Try using wc_InitRsaKey(&key) and not the _ex version. If you use _ex make sure your HEAP_HINT is NULL and devId is INVALID_DEVID.
What key size are you trying to use with your RSA server certificate? Our library enables RSA 2048-bit by default and you would need to define a higher FP_MAX_BITS (key size * 2). Can send me the public self-signed server certificate that you are using?
Also make sure you have HAVE_TLS_EXTENSIONS and HAVE_SUPPORTED_CURVES defined if using ECDHE and TLS.
If you are able to make the server external that would help us be able to test against it. Feel free to reach out to us at support@wolfssl.com if you don't want any of your server information public on the forum.
You have a great holiday as well!
Thanks,
David Garske, wolfSSL