To clarify: disabling USE_FAST_MATH is what caused the error.

I also tested various permutations of enabling/disabling the following:
FP_MAX_BITS_ECC (disabled or enabled with a few different sizes, as you suggested)
TFM_ASM
TFM_TIMING_RESISTANT
TFM_ECC521
ECC_TIMING_RESISTANT

None seemed to have an effect on success or failure, it always came down to: failure with USE_FAST_MATH disabled, success with it enabled.

Also, another question: what are any advantages/disadvantages/tradeoffs in using ALT_ECC_SIZE? Enabling/disabling this didn't seem to affect either code size or performance times.

Updating to 3.9.8 did fix the issue. I was able to run the tests successfully.

However, while testing performance/optimization, I rebuilt after disabling TFM. Running the test once again gives a -1023 error.

After I first pulled 3.9.8 I thought I successfully ran it with and without TFM. Since then I've updated the chip library from LPCopen 2.12 to 2.18, so maybe this is causing some issue? (I could go back and double check on 2.12 but downgrading would be a pain)

Edit: I tried disabling just TFM_ECC521 (as opposed to disabling USE_FAST_MATH, ALT_ECC_SIZE, and every TFM_ entry), and the test worked. Maybe I just missed one of the related #defines?

David,

ECC_SHAMIR was already enabled by default in the IDE/LPCXPRESSO user_settings file.
ECC_TIMING_RESSISTANT was not defined; adding this in had no effect (default curve success, P-521 curve failure).
TFM_ECC521 was also not defined; adding this in caused a different failure, even with the default curve:

From inside test_ecc, the failure occurred here, with output -1024:

    ret = wc_ecc_check_key(&userA);
    if (ret != 0)
        return -1024;

Here is where the failure occurs inside wc_ecc_check_key (err returns -1):

    /* make sure point is actually on curve */
    if (err == MP_OKAY)
        err = ecc_is_point(key->dp, &key->pubkey, &prime);

I'll look into trying out 3.9.8.

My project involves implementing functions from wolfCrypt on an NXP LPC1837 MCU.

I'm able to build and run the wolf_example project in the IDE/LPCXPRESSO directory. Running the test with default settings is a success, but if I attempt to add in support for the P-521 curve (#define HAVE_ECC521 in user_settings.h) the test fails, with error code -1023.

Separate from the wolf_example project, I've also tested wolfCrypt within my own project by including just the necessary .c and .h files for the functions I need. I've done a bit of digging to figure out exactly where the error occurs:

Hash verification fails at line 2469 of ecc.c

2467   /* does v == r */
2468   if (err == MP_OKAY) {
2469       if (mp_cmp(&v, r) == MP_EQ)
2470           *stat = 1;
2471   }

This happens within line 6261 of test.c, causing the return at 6266

6261    ret = wc_ecc_verify_hash(sig, x, (byte*)vector->msg, (word32)vector->msgLen, &verify, userA);
6262    if (ret != 0)
6263        return -1021;
6264
6265    if (verify != 1)
6266        return -1023;

which is called within line 6598 of test.c

6597        ecc521.curveName = "nistp521";
6598        ret = ecc_test_raw_vector(&ecc521, &userA, sig, sizeof(sig));
6599        if (ret < 0) {
6600            return ret;
6601        }

To utilize the P-521 curve, are the additional steps needed besides defining HAVE_ECC521?
If not, what else might be causing this issue?