Do you know where the SSL handshake is getting stuck?  Have you turned on wolfSSL debug messages?  You can do so by adding DEBUG_WOLFSSL to your preprocessor flags, then calling wolfSSL_Debugging_ON() / wolfSSL_Debugging_OFF() from your code.

By default wolfSSL uses send() and recv() in its default I/O callbacks.  You should be able to set a timeout value on your socket when you initially create it, using setsockopt().

Yes, you can use non-blocking I/O without defining WOLFSSL_CALLBACKS.  woflSSL can use either blocking or non-blocking I/O.  See the API reference for wolfSSL_connect(), wolfSSL_accept(), wolfSSL_read(), or wolfSSL_write() for more info on how each works when non-blocking I/O is used.

Our example client and server are also set up to do non-blocking I/O when the "-N" command line option is used.  Feel free to look at client.c or server.c for example usage.

Best Regards,
Chris

We have a problem during the client-server communication, where the server doesn't complete the SSL handshake. This cause the connect function to be stuck forever and it never returns.

We want to have a (blocking) connect function that we can call with a timeout, and if it doesn't able to complete the connection during that timeout, it simply returns with an error. Can I do that at all?

Because we failed to do the above, we thought of using the non-blocking connect call. The problem was, we couldn't compile it with WOLFSSL_CALLBACKS set.  Can I do that without defining WOLFSSL_CALLBACKS?

Thanks  again!

It looks like WOLFSSL_CALLBACKS doesn't currently support Windows, as like you noted, it uses <sys/time.h>.  Will you need to use WOLFSSL_CALLBACKS?  The handshake callback (WOLFSSL_CALLBACKS) is intended for embedded systems for debugging support.  That is, when a good debugger isn't available and sniffing is impractical.  It basically allows a trace of the handshake embedded SSL.

Regards,
Chris