I agree that TLS using certificates are usually a more robust form of authentication. However, it is also very costly. I need a lighter way which only focus on protecting the authentication. I'm not trying to reinvent anything though; just make use of common RSA encryption with manually distributed key pairs.

The API doesn't need to be similar to openssl, and in fact I like C++ style Taocrypt better than the C-hackish openssl. The files and parameter values used need to stay compatible with openssl though (although it isn't strictly necessary).

I need to transfer an encrypted password from the client to the server with the simplest possible handshake. It would be preferable if the solution could stay openssl compatible at the same time. I think fixing the decoder shouldn't be all that much work actually, but I would indeed appreciate some help.

The main reason is that it is bundled with MySQL and I'm limited to what's contained within the next release of MySQL. Are you saying that the only way to import a public RSA key in DER or PEM format is to rewrite the decoder class?

Cheers,
Kristofer

I'm trying to follow the examples provided to load a public RSA key. I generated the key as:

openssl genrsa -out my_priv_key.pem 4096
openssl rsa -in my_priv_key.pem -out my_pub_key.pem -pubout

Then converted the PEM files to DER files like this:

openssl rsa -in  my_priv_key.pem  -inform PEM -out  my_priv_key.der -outform DER
openssl rsa -in  my_priv_key.pem -pubout -out my_pub_key.der -outform DER

Parsing the private key works fine and I can successfully extract the public key components from this key to encrypt and decrypt a message (using   rsaPub.SetModulus(n); rsaPub.SetPublicExponent(e)), but when I try to do:

TaoCrypt::RSA_PublicKey rsaPub(public_key_src);
TaoCrypt::PK_Lengths lengths(rsaPub.GetModulus());
printf("Pub key: FixedCiphertextLength= %u\n",lengths.FixedCiphertextLength());

I get the result:
Pub key: FixedCiphertextLength= 0

The failure seems to happen in RSA_Public_Decode::Decode() :

// Decode a BER encoded RSA Public Key
void RSA_Public_Decoder::Decode(RSA_PublicKey& key)
{
    ReadHeader();
    if (source_.GetError().What()) return;

    // public key
    key.SetModulus(GetInteger(Integer().Ref()));  <---- Fails
    key.SetPublicExponent(GetInteger(Integer().Ref()));
}

and more specifically here:

void Integer::Decode(Source& source)
{
    byte b = source.next();
    if (b != INTEGER) {
        source.SetError(INTEGER_E);  <-- exit path
        return;
    }
[..]

What is the proper way to store the public key so that it can be parsed correctly?

Regards,
Kristofer Pettersson