Topic: [SOLVED] MP_EXPTMOD_E during SSL handshake
we came across the problem that during the SSL Handshake (using ECDHE-RSA-AES128-GCM-SHA256)
WolfSSL is returning with VERIFY_SIGN_ERROR (-330). The Error occurs in one test environment while on
another everything works as expected. In both environments WOLFSSL is used on the client side. While the servers SSL Stack is unknown.
We tracked the error down further and found that a call to mp_exptmod() results in MP_EXPTMOD_E
That happens during processing server key exchange (which is printed out right before)
Digging further down we found the following condition to be the reason for that behavior:
(TAG 3.6.9) tfm.c:1206 if (P->used > (FP_SIZE/2))
We have logged the parameters from both of the machines:
In our test environment (Client compiled for and runs on iOS Simulator 64Bit) no1 the parameters are
FP_SIZE/2: 36 --> sizeof(fp_digit) == 8 (64Bit mode)
Result: handshake succeeds
In the second test envioronment (Client compiled for and runs on iOS Device 64Bit) the condition matches with the following values:
FP_SIZE/2: 68 --> sizeof(fp_digit) == 4 (compiled for 32Bit mode, I think ARM64 is not supported)
Result: VERIFY_SIGN_ERROR due tue MP_EXPTMOD_E due to FP_VAL returned by fp_exptmod
We need some suggestion on how to proceed from here. We are unsure if the problem is due to a client or server problem.
We use wolfssl 3.6.9 in an iOS environment and have build with the following settings:
#define SIZEOF_LONG_LONG 8
Thank you very much!