It's kind of strange that that function isn't being documented. From what I can tell, it should be documented on this page here: https://www.wolfssl.com/doxygen/wolfssl_2ssl_8h.html. In regards to the actual explanation of what that function does, let me share the snippet that should be describing the function on that page.
Allows caller to set the Public Key Callback for RSA Verification.
The callback should return the number of plaintext bytes for success or
< 0 for an error. The ssl and ctx pointers are available for the user’s
convenience. sig is the signature to verify and sigSz denotes the length
of the signature. out should be set to the beginning of the verification
buffer after the decryption process and any padding. keyDer is the RSA
Public key in ASN1 format and keySz is the length of the key in bytes.
An example callback can be found wolfssl/test.h myRsaVerify().
The function "wolfSSL_CTX_SetRsaVerifyCb" should not return anything on its own, as it's return type is void.
For future reference, when these functions are not being described in the documentation, many wolfSSL API have descriptions located within the wolfSSL library itself, at the following location: <wolfssl-root>/doc/dox_comments/header_files/.
Please let me know if there was anything I missed or if you'd like any further information.