You will want to use SP math rather than Fast math to increase your speed.  Replace USE_FAST_MATH with the following:
WOLFSSL_SP_MATH_ALL
WOLFSSL_SP_ASM
Let me know if you still have speed issues.

Thanks,
Kareem

Any idea why removing GCM_SMALL define did not have any impact on speed? Anything else should be removed?

Currently my defines look like this:

CUSTOM_RAND_GENERATE_SEED
NO_WRITEV
NO_WOLFSSL_DIR
TFM_TIMING_RESISTANT
ECC_TIMING_RESISTANT
WC_RSA_BLINDING
HAVE_AESCCM
HAVE_AESGCM
HAVE_ECC
HAVE_AES_CBC
WOLFSSL_AES_128
WOLFSSL_AES_256
ECC_USER_CURVES
HAVE_HKDF
HAVE_FFDHE_2048
WC_RSA_PSS
WOLFSSL_ENCRYPTED_KEYS
HAVE_TLS_EXTENSIONS
HAVE_SUPPORTED_CURVES
WOLFSSL_SHA384
WOLFSSL_SHA512
WOLFSSL_TLS13
USE_FAST_MATH
ALT_ECC_SIZE

Regards,

Alex

If your platform doesn't support hardware acceleration for AES, I would recommend trying Chacha/Poly.  Yes, you should remove your fast math define.
You are correct that ARMASM won't work with ARMv5, but you should still be able to use SP math assembly which I would definitely recommend.

Thanks,
Kareem

I am working on armv5 chip, so WOLFSSL_ARMASM will not work for me I guess. I tried removing GCM_SMALL and it had no impact on the speed. What cipher suite would you recommend for faster processing speed? Also , to use SP math do in eed to remove fast math define?

Regards,

Alex

Glad to hear you were able to resolve your issue.  What platform/architecture are you on?  Which cipher suite are you using?
In general, I'd recommend using SP math over fast math, and SP math assembly if your platform supports it.
Check out our user settings template here for more info on our SP math and other settings: https://github.com/wolfSSL/wolfssl/blob … template.h
I would also enable ARM assembly with WOLFSSL_ARMASM, assuming you're on ARM.
You will want to disable small AES-GCM if possible as it will have a significant speed impact (GCM_SMALL).

Thanks,
Kareem

Switching to server TLS 1.3 from TLS 1.2 resolves the problem I had, but it makes processing each connection request 4 or 5 times longer. What is the best way to optimize for speed? Right now each request takes 2 seconds to process, which will create a jerky performance for users.

Regards,

Alex

To answer your questions:

What platform and OS are you using?
I am using embedded RTOS
How much stack/heap are you giving to wolfSSL?
I believe stack and heap should be sufficient since website is running normally after browser restart. Also during reboot all the wolfssl objects are released.
What kind of webpage are you serving with wolfSSL?
It a webpage of user interface that constantly sending ajax requests and gets back some data.
Is it something that would spawn multiple connections?
There could be several request pending, but not to much.
Does closing your Chrome tab and opening a new one resolve the issue, or does it require a full browser restart?
Closing reopening tab is not working. Only full restart of the browser does.
It does seem like Chrome tries to connect to previous connection since netstat list many TIME_WAIT ports trying to connect to server after server is restarted once.

After first restart of the server the browser still connects to website however every second wolfssl_accept gives alert error as shown in previous posts. After second restart, browsers can't connect to website and just displays ssl_connection_error, while every wolfssl_accept throws an error.

Regards,

Alex

What platform and OS are you using?  How much stack/heap are you giving to wolfSSL?
What kind of webpage are you serving with wolfSSL?  Is it something that would spawn multiple connections?
Does closing your Chrome tab and opening a new one resolve the issue, or does it require a full browser restart?

If this information is too sensitive to share in a public forum, feel free to contact us at support@wolfssl.com

Thanks,
Kareem

After updating to 5.0, I still have the same problem. After rebooting a server once and not touching the browser following is the log of the error:

WOLFSSL MSG: wolfSSL Entering SSL_accept()
WOLFSSL MSG: Client attempting to connect with different version
WOLFSSL MSG: growing input buffer
WOLFSSL MSG: received record layer msg
WOLFSSL MSG: got HANDSHAKE
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsg()
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsgType
WOLFSSL MSG: processing client hello
WOLFSSL MSG: wolfSSL Entering DoClientHello
WOLFSSL MSG: Client wants to resume session
WOLFSSL MSG: Matched No Compression
WOLFSSL MSG: Adding signature algorithms extension
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Secure Renegotiation extension received
WOLFSSL MSG: Supported Groups extension received
WOLFSSL MSG: Point Formats extension received
WOLFSSL MSG: Session Ticket extension received
WOLFSSL MSG: ALPN extension received
WOLFSSL MSG: Certificate Status Request extension received
WOLFSSL MSG: Signature Algorithms extension received
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Key Share extension received
WOLFSSL MSG: PSK Key Exchange Modes extension received
WOLFSSL MSG: Skipping Supported Versions - already processed
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Found a session match
WOLFSSL MSG: Session valid
WOLFSSL MSG: wolfSSL Entering MatchSuite
WOLFSSL MSG: wolfSSL Entering VerifyServerSuite
WOLFSSL MSG: Requires RSA
WOLFSSL MSG: Requires AEAD
WOLFSSL MSG: Verified suite validity
WOLFSSL MSG: wolfSSL Leaving DoClientHello, return 0
WOLFSSL MSG: Shrinking input buffer
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsgType(), return 0
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsg(), return 0
WOLFSSL MSG: accept state ACCEPT_CLIENT_HELLO_DONE
WOLFSSL MSG: accept state ACCEPT_FIRST_REPLY_DONE
WOLFSSL MSG: wolfSSL Entering SendServerHello
WOLFSSL MSG: Point Formats extension to write
WOLFSSL MSG: wolfSSL Leaving SendServerHello, return 0
WOLFSSL MSG: accept state SERVER_HELLO_SENT
WOLFSSL MSG: accept state CERT_SENT
WOLFSSL MSG: accept state CERT_STATUS_SENT
WOLFSSL MSG: accept state KEY_EXCHANGE_SENT
WOLFSSL MSG: accept state CERT_REQ_SENT
WOLFSSL MSG: accept state SERVER_HELLO_DONE
WOLFSSL MSG: accept state  ACCEPT_SECOND_REPLY_DONE
WOLFSSL MSG: accept state  TICKET_SENT
WOLFSSL MSG: accept state  CHANGE_CIPHER_SENT
WOLFSSL MSG: wolfSSL Entering SendFinished
WOLFSSL MSG: wolfSSL Leaving SendFinished, return 0
WOLFSSL MSG: accept state ACCEPT_FINISHED_DONE
WOLFSSL MSG: received record layer msg
WOLFSSL MSG: got ALERT!
WOLFSSL MSG: wolfSSL error occurred, error = -132
WOLFSSL MSG: wolfSSL Entering SSL_get_error
WOLFSSL MSG: wolfSSL Leaving SSL_get_error, return -132
WOLFSSL MSG: wolfSSL Entering SSL_get_error
WOLFSSL MSG: wolfSSL Leaving SSL_get_error, return -132
WOLFSSL MSG: wolfSSL Entering wolfSSL_ERR_error_string_n
WOLFSSL MSG: Error buffer too short, truncating
WOLFSSL MSG: wolfSSL Entering ERR_error_string
WOLFSSL MSG: wolfSSL Entering SSL_shutdown()
WOLFSSL MSG: wolfSSL Entering SendAlert
WOLFSSL MSG: Embed Send error
WOLFSSL MSG:    Connection aborted
WOLFSSL MSG: wolfSSL Leaving SendAlert, return -308
WOLFSSL MSG: wolfSSL error occurred, error = -308
WOLFSSL MSG: wolfSSL Entering SSL_free
WOLFSSL MSG: CTX ref count not 0 yet, no free
WOLFSSL MSG: wolfSSL Leaving SSL_free, return 0
WOLFSSL MSG: wolfSSL Entering SSL_new
WOLFSSL MSG: wolfSSL Leaving SSL_new, return 0

After rebooting twice, I get the following behaviour:

WOLFSSL MSG: wolfSSL Entering SSL_accept()
WOLFSSL MSG: Client attempting to connect with different version
WOLFSSL MSG: growing input buffer
WOLFSSL MSG: received record layer msg
WOLFSSL MSG: got HANDSHAKE
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsg()
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsgType
WOLFSSL MSG: processing client hello
WOLFSSL MSG: wolfSSL Entering DoClientHello
WOLFSSL MSG: Client wants to resume session
WOLFSSL MSG: Matched No Compression
WOLFSSL MSG: Adding signature algorithms extension
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Secure Renegotiation extension received
WOLFSSL MSG: Supported Groups extension received
WOLFSSL MSG: Point Formats extension received
WOLFSSL MSG: Session Ticket extension received
WOLFSSL MSG: ALPN extension received
WOLFSSL MSG: Certificate Status Request extension received
WOLFSSL MSG: Signature Algorithms extension received
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Key Share extension received
WOLFSSL MSG: PSK Key Exchange Modes extension received
WOLFSSL MSG: Skipping Supported Versions - already processed
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Unknown TLS extension type
WOLFSSL MSG: Session lookup for resume failed
WOLFSSL MSG: wolfSSL Entering MatchSuite
WOLFSSL MSG: wolfSSL Entering VerifyServerSuite
WOLFSSL MSG: Requires RSA
WOLFSSL MSG: Requires AEAD
WOLFSSL MSG: Verified suite validity
WOLFSSL MSG: wolfSSL Leaving DoClientHello, return 0
WOLFSSL MSG: Shrinking input buffer
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsgType(), return 0
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsg(), return 0
WOLFSSL MSG: accept state ACCEPT_CLIENT_HELLO_DONE
WOLFSSL MSG: accept state ACCEPT_FIRST_REPLY_DONE
WOLFSSL MSG: wolfSSL Entering SendServerHello
WOLFSSL MSG: Point Formats extension to write
WOLFSSL MSG: wolfSSL Leaving SendServerHello, return 0
WOLFSSL MSG: accept state SERVER_HELLO_SENT
WOLFSSL MSG: wolfSSL Entering SendCertificate
WOLFSSL MSG: wolfSSL Leaving SendCertificate, return 0
WOLFSSL MSG: accept state CERT_SENT
WOLFSSL MSG: wolfSSL Entering SendCertificateStatus
WOLFSSL MSG: wolfSSL Leaving SendCertificateStatus, return 0
WOLFSSL MSG: accept state CERT_STATUS_SENT
WOLFSSL MSG: wolfSSL Entering SendServerKeyExchange
WOLFSSL MSG: Using ephemeral ECDH
WOLFSSL MSG: wolfSSL Entering EccMakeKey
WOLFSSL MSG: wolfSSL Leaving EccMakeKey, return 0
WOLFSSL MSG: Trying RSA private key
WOLFSSL MSG: wolfSSL Entering GetAlgoId
WOLFSSL MSG: wolfSSL Entering GetObjectId()
WOLFSSL MSG: Using RSA private key
WOLFSSL MSG: wolfSSL Entering RsaSign
WOLFSSL MSG: wolfSSL Using RSA PSS padding
WOLFSSL MSG: wolfSSL Leaving RsaSign, return 0
WOLFSSL MSG: wolfSSL Entering VerifyRsaSign
WOLFSSL MSG: wolfSSL Using RSA PSS un-padding
WOLFSSL MSG: wolfSSL Leaving VerifyRsaSign, return 0
WOLFSSL MSG: wolfSSL Entering SendHandshakeMsg
WOLFSSL MSG: Embed Send error
WOLFSSL MSG:    Connection aborted
WOLFSSL MSG: wolfSSL Leaving SendServerKeyExchange, return -308
WOLFSSL MSG: wolfSSL error occurred, error = -308
WOLFSSL MSG: wolfSSL Entering SSL_get_error
WOLFSSL MSG: wolfSSL Leaving SSL_get_error, return -308
WOLFSSL MSG: wolfSSL Entering SSL_get_error
WOLFSSL MSG: wolfSSL Leaving SSL_get_error, return -308
WOLFSSL MSG: wolfSSL Entering wolfSSL_ERR_error_string_n
WOLFSSL MSG: Error buffer too short, truncating
WOLFSSL MSG: wolfSSL Entering ERR_error_string
WOLFSSL MSG: wolfSSL Entering SSL_shutdown()
WOLFSSL MSG: wolfSSL Leaving SSL_shutdown(), return -1
WOLFSSL MSG: wolfSSL Entering SSL_free
WOLFSSL MSG: CTX ref count not 0 yet, no free
WOLFSSL MSG: wolfSSL Leaving SSL_free, return 0
WOLFSSL MSG: wolfSSL Entering SSL_new
WOLFSSL MSG: wolfSSL Leaving SSL_new, return 0
WOLFSSL MSG: wolfSSL Entering SSL_set_fd
WOLFSSL MSG: wolfSSL Entering SSL_set_read_fd
WOLFSSL MSG: wolfSSL Leaving SSL_set_read_fd, return 1
WOLFSSL MSG: wolfSSL Entering SSL_set_write_fd
WOLFSSL MSG: wolfSSL Leaving SSL_set_write_fd, return 1

Regards,

Alex

3.15.3 is extremely outdated and unsupported.  Please update to 5.0 and let me know if it fixes the issue for you.

To answer your questions:

1) hard refresh does not work
2) I am running version 3.15.3

A couple of questions:
1) Does a hard refresh (usually Ctrl+F5) show the same issue in your browser?
2) What version of wolfSSL are you using?  Do you still see the issue with the latest wolfSSL, 5.0?

Thanks,
Kareem

Thank you for clarification. I am running wolfssl as a server and a client just a chrome browser.

My current problem is that any time I reset the server browser start failing to exhange certificate with the server resuilting in following sequence:

WOLFSSL MSG: wolfSSL Entering SSL_accept()
WOLFSSL MSG: Client attempting to connect with different version
WOLFSSL MSG: growing input buffer
WOLFSSL MSG: received record layer msg
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsg()
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsgType
WOLFSSL MSG: processing client hello
WOLFSSL MSG: wolfSSL Entering DoClientHello
WOLFSSL MSG: Client wants to resume session
WOLFSSL MSG: Matched No Compression
WOLFSSL MSG: Adding signature algorithms extension
WOLFSSL MSG: Secure Renegotiation extension received
WOLFSSL MSG: Supported Groups extension received
WOLFSSL MSG: Point Formats extension received
WOLFSSL MSG: Session Ticket extension received
WOLFSSL MSG: ALPN extension received
WOLFSSL MSG: Certificate Status Request extension received
WOLFSSL MSG: Signature Algorithms extension received
WOLFSSL MSG: Key Share extension received
WOLFSSL MSG: PSK Key Exchange Modes extension received
WOLFSSL MSG: Supported Versions extension received
WOLFSSL MSG: Session lookup for resume failed
WOLFSSL MSG: wolfSSL Entering MatchSuite
WOLFSSL MSG: wolfSSL Entering VerifyServerSuite
WOLFSSL MSG: Requires RSA
WOLFSSL MSG: Verified suite validity
WOLFSSL MSG: wolfSSL Leaving DoClientHello, return 0
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsgType(), return 0
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsg(), return 0
WOLFSSL MSG: accept state ACCEPT_CLIENT_HELLO_DONE
WOLFSSL MSG: accept state ACCEPT_FIRST_REPLY_DONE
WOLFSSL MSG: wolfSSL Entering SendServerHello
WOLFSSL MSG: Point Formats extension to write
WOLFSSL MSG: wolfSSL Leaving SendServerHello, return 0
WOLFSSL MSG: accept state SERVER_HELLO_SENT
WOLFSSL MSG: wolfSSL Entering SendCertificate
WOLFSSL MSG: wolfSSL Leaving SendCertificate, return 0
WOLFSSL MSG: accept state CERT_SENT
WOLFSSL MSG: wolfSSL Entering SendCertificateStatus
WOLFSSL MSG: wolfSSL Leaving SendCertificateStatus, return 0
WOLFSSL MSG: accept state CERT_STATUS_SENT
WOLFSSL MSG: wolfSSL Entering SendServerKeyExchange
WOLFSSL MSG: Using ephemeral ECDH
WOLFSSL MSG: wolfSSL Entering EccMakeKey
WOLFSSL MSG: wolfSSL Leaving EccMakeKey, return 0
WOLFSSL MSG: wolfSSL Entering RsaSign
WOLFSSL MSG: wolfSSL Using RSA PSS padding
WOLFSSL MSG: wolfSSL Leaving RsaSign, return 0
WOLFSSL MSG: wolfSSL Entering VerifyRsaSign
WOLFSSL MSG: wolfSSL Using RSA PSS un-padding
WOLFSSL MSG: wolfSSL Leaving VerifyRsaSign, return 0
WOLFSSL MSG: Embed Send error
WOLFSSL MSG:    General error
WOLFSSL MSG: wolfSSL Leaving SendServerKeyExchange, return -5307
WOLFSSL MSG: wolfSSL error occurred, error = -5307
WOLFSSL MSG: wolfSSL Entering SSL_get_error

I do wolfssl_cleanup and free ctx object as well as close all listening sockets before reseting the server, but it seems the browser will not accept certificate after reset of the server, unless I close and reopen the browser.

Any ideas on how to force or notify browser that the old connection is closed? Or anything else I am missing?

Regards,

Alex

"Client attempting to connect with different version" means the client and server's TLS versions are not matching.  What TLS version is your client using?
If for example your client is using TLS 1.0, you will need to define WOLFSSL_ALLOW_TLSV10.  If your client is using TLS 1.2, it may help to undefine WOLFSSL_TLS13.

Thanks,
Kareem

Here the debug messages I get:

WOLFSSL MSG: wolfSSL Entering SSL_accept()
WOLFSSL MSG: Client attempting to connect with different version
WOLFSSL MSG: growing input buffer
WOLFSSL MSG: received record layer msg
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsg()
WOLFSSL MSG: wolfSSL Entering DoHandShakeMsgType
WOLFSSL MSG: processing client hello
WOLFSSL MSG: wolfSSL Entering DoClientHello
WOLFSSL MSG: Client wants to resume session
WOLFSSL MSG: Matched No Compression
WOLFSSL MSG: Adding signature algorithms extension
WOLFSSL MSG: Secure Renegotiation extension received
WOLFSSL MSG: Supported Groups extension received
WOLFSSL MSG: Point Formats extension received
WOLFSSL MSG: Session Ticket extension received
WOLFSSL MSG: ALPN extension received
WOLFSSL MSG: Certificate Status Request extension received
WOLFSSL MSG: Signature Algorithms extension received
WOLFSSL MSG: Key Share extension received
WOLFSSL MSG: PSK Key Exchange Modes extension received
WOLFSSL MSG: Supported Versions extension received
WOLFSSL MSG: Found a session match
WOLFSSL MSG: Session valid
WOLFSSL MSG: wolfSSL Entering MatchSuite
WOLFSSL MSG: wolfSSL Entering VerifyServerSuite
WOLFSSL MSG: Requires RSA
WOLFSSL MSG: Verified suite validity
WOLFSSL MSG: wolfSSL Leaving DoClientHello, return 0
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsgType(), return 0
WOLFSSL MSG: wolfSSL Leaving DoHandShakeMsg(), return 0
WOLFSSL MSG: accept state ACCEPT_CLIENT_HELLO_DONE
WOLFSSL MSG: accept state ACCEPT_FIRST_REPLY_DONE
WOLFSSL MSG: wolfSSL Entering SendServerHello
WOLFSSL MSG: Point Formats extension to write
WOLFSSL MSG: wolfSSL Leaving SendServerHello, return 0
WOLFSSL MSG: accept state SERVER_HELLO_SENT
WOLFSSL MSG: accept state CERT_SENT
WOLFSSL MSG: accept state CERT_STATUS_SENT
WOLFSSL MSG: accept state KEY_EXCHANGE_SENT
WOLFSSL MSG: accept state CERT_REQ_SENT
WOLFSSL MSG: accept state SERVER_HELLO_DONE
WOLFSSL MSG: accept state  ACCEPT_SECOND_REPLY_DONE
WOLFSSL MSG: accept state  TICKET_SENT
WOLFSSL MSG: accept state  CHANGE_CIPHER_SENT
WOLFSSL MSG: wolfSSL Entering SendFinished
WOLFSSL MSG: wolfSSL Leaving SendFinished, return 0
WOLFSSL MSG: accept state ACCEPT_FINISHED_DONE
WOLFSSL MSG: received record layer msg
WOLFSSL MSG: got ALERT!
WOLFSSL MSG: Got alert
WOLFSSL MSG: wolfSSL error occurred, error = 20
WOLFSSL MSG: wolfSSL error occurred, error = -313
WOLFSSL MSG: wolfSSL Entering SSL_get_error
WOLFSSL MSG: wolfSSL Leaving SSL_get_error, return -313

What does line "Client attempting to connect with different version" means?

Any obvious reasons why I would get intermittent Alert 20 messages?

Any help would be appreciated.

Thanks