Hello Brett,
thank you for your response.
For SHA variants, have you looked at the WOLFSSL_HASH_FLAGS option? This should do just what you are looking for - it will add flags to the context indicating whether or not the struct is a copy or the original. I imagine that using this combined with adding a refcount to your custom context should be enough for your cryptocb to know when to free the custom context when a finalization operation is detected?
I actually noticed these flags five minutes after I made the post.
The issue here is that I can use the "is copy" flag to know if the wolfSSL-SHA object is a copy so I don't free my resources, but there are some occasions where the getHash function is called (which does a copy + finalize of the hash object to get the digest) but the original might not call the finalize function. So my idea of "free on finalize" wouldn't work out if the original never finalizes.
My current solution is just patching the wolfSSL code to notify me on init/copy/free plus reference counter which seems to not leak any resources and not run into issues, yet. For testing purposes I'm still doing software hashing and the HSM ownership stuff is running in parallel just to test the sanity of everything. I'll try to actually feed the HSM on Monday.
I'll probably go with the patching + ref counter approach with every other domain (AES, signing and whatever else comes up).
Regarding wolfHSM:
My colleagues are actually evaluating it but have nothing that I can use right now, so I'm stuck with etas for the time being so I can get the proof-of-concept stuff out of the door.
Also, I'm currently in contact via mail with someone from your German colleagues who might be able to clarify some stuff.