• Registered: 2017-07-02
  • Posts: 24

Topic: [SOLVED] Decrypt problem in WolfSSL

Hello:

I 'm using the wolfSSL library to secure an MQTT connection with AWS IoT using mutual authentication. A device publishes messages and subscribes to a couple of topics.
So far, I don't have any problem publishing, but eventually, I have decryption problems when receiving a message.

So basically everything works fine until at some point I get this message in the console;
  .....SendAlert.....2 20
which comes from SendAlert(), and means severity: 2, type:20

Debugging the issue, I found problem happens in Decrypt() (internal.c), in case wolfssl_aes_gcm which calls  function wc_AesGcmDecrypt() returning VERIFY_MAC_ERROR -305

Above function is called from ProcessReply() which will return DECRYPT_ERROR  -312 ; which in turns is called by wolfSSL_read() from mqtt_packet_read() function.

Here is the debug info printed when using #define DEBUG_WOLFSSL option in settings.

wolfSSL Entering AesGcmDecrypt
.....SendAlert.....2 20
growing output buffer

wolfSSL Entering AesGcmEncrypt
Shrinking output buffer

Decrypt failed
wolfSSL error occurred, error = -305
wolfSSL error occurred, error = -312
wolfSSL Leaving wolfSSL_read_internal(), return -312
wolfSSL Entering SSL_get_error
wolfSSL Leaving SSL_get_error, return -312
wolfSSL Entering wolfSSL_read()
wolfSSL Entering wolfSSL_read_internal()
wolfSSL Entering ReceiveData()
User calling wolfSSL_read in error state, not allowed
wolfSSL Leaving wolfSSL_read_internal(), return -312

Additionally, from this point on, the publishing is not longer working as the stack remains in an internal error state, so it basically blocks the device for further communication. (I see the publishing messages being sent - also see tcp packes in wireshark - but it seems the server discards these messages as they probably have something wrong...)

My questions are:
1) what could cause this error? I tried with different messages and I can't see a relation with a message type.
2) in case I can't avoid this error; how I can recover from this situation? What should I do to lose the received packet but continue with rest of tasks?

Thanks in advance
Regards;
Gus

Share

  • Registered: 2015-10-07
  • Posts: 415

Re: [SOLVED] Decrypt problem in WolfSSL

Hi Gus,

Typically if things break-down after the connection has already been running fine indicates to me some memory issue either with corruption or memory leak. I recommend adding some memory checking to measure heap use and also stack use. We have some hooks for monitoring heap that can be enabled using `WOLFSSL_TRACK_MEMORY`. This will add tracking on XMALLOC/XFREE and display a result on wolfSSL_Cleanup() or `ShowMemoryTracker()`. Measuring stack is typically RTOS specific. What OS / Micro-controller are you using? Are you using any hardware crypto acceleration?

A typical recovery for a connection error would be to call `wolfSSL_shutdown(ssl)`, `wolfSSL_free(ssl)` and `close()` the socket. This will cleanup everything and allow you to reconnect again just as you did before. If you are having a memory issue I suspect you'll see problems reconnecting as well. We have a collection of TLS examples here: https://github.com/wolfSSL/wolfssl-exam … master/tls

Thanks,
David Garske, wolfSSL

Share

3 Reply by gussabina (edited by gussabina 2017-09-18 12:53:34)

  • Registered: 2017-07-02
  • Posts: 24

Re: [SOLVED] Decrypt problem in WolfSSL

Hello David:

Thank you very much for your support.

Answering your questions, I'm using the Atmel Zero touch AWS IoT kit (http://www.atmel.com/applications/iot/aws-zero-touch-secure-provisioning-platform/default.aspx?tab=overview ) as a base code. The project is using FreeRTOS and the mcu is Atmel ATSAM4E (with AES) and the kit comes with ATECC508 Crypto chip. (Original code was provided by ATSAMG55 but I ported to my mcu).
Honestly, I'm not sure if Encryption is done by Hardware or Software; how can I check this?

Thanks for the input; I will try to check the memory issue. I will continue investigating on this direction. Any other input, it will be much appreciated.

UPDATE: I tried to search for 'WOLFSSL_TRACK_MEMORY' but it is not there... In settings.h, the configuration define is 'ATMEL_AWS_WOLFSSL' with following configuration;

#if defined(ATMEL_AWS_WOLFSSL)
    #define SINGLE_THREADED
    #define HAVE_ECC
    #define HAVE_AESGCM
    #define HAVE_PK_CALLBACKS
    #define NO_FILESYSTEM
    #define NO_PSK
    #define NO_OLD_TLS
    #define NO_WRITEV
    #define NO_WOLFSSL_DIR
    #define NO_DEV_RANDOM
    #define NO_DSA
    #define NO_HC128
    #define NO_RABBIT
    #define NO_MD2
    #define NO_MD4
    #define NO_MD5
    #define NO_SHA
    #define NO_RC4
    #define NO_DES3
    #define NO_PWDBASED
    #define NO_SKID
    #define NO_SESSION_CACHE
    #define WOLFSSL_USER_IO
    #define WOLFSSL_STATIC_DH
    #define WOLFSSL_CERT_GEN
    #define WOLFSSL_SMALL_STACK
    //#define DEBUG_WOLFSSL
    #define NO_WOLFSSL_SERVER    
#endif

Regards;
Gus

Share

  • Registered: 2017-07-02
  • Posts: 24

Re: [SOLVED] Decrypt problem in WolfSSL

Hello:

I reviewed the potential memory leaks but so far I'm using malloc() in two places;  to create a socket and a global structure that keeps all MQTT info in one place. I'm not creating those objects more than once...

What I believe I could have is a memory corruption issue; either in the stack or in some data buffer being accessed simultaneously from different tasks...

BTW, I realized in the configuration, I'm using #define SINGLE_THREADED which is not actually the case since I'm using FreeRTOS. Should I use a different one? Is there any restriction/condition for wolfSSL when used with FreeRTOS?

Thanks
Gus

Share

  • Registered: 2015-10-07
  • Posts: 415

Re: [SOLVED] Decrypt problem in WolfSSL

Hi Gussabina,

If you are using FreeRTOS you can define `FREERTOS` instead of `SINGLE_THREADED` to enable mutex protection in the wolfSSL library.

The wolfMQTT library on the other-hand does not support multi-threads. To solve that you have a couple options:
1. Have one thread dedicated to handling the RX and TX of publish messages. Define `WOLFMQTT_NONBLOCK` and if you get back the `MQTT_CODE_CONTINUE` response code then it means no RX data is available. You can then use/check a FreeRTOS queue for any messages to TX.
2. Use a mutex to protect access to the socket between two threads. You'll also need to use non-blocking mode so you aren't blocked on the socket read.

It sounds like the eventual failure might be due to multiple threads trying to rx/tx publish messages at the same time. This would get the data in the socket out of sync and probably cause the decrypt error.

Thanks,
David Garske, wolfSSL

Share

6 Reply by gussabina (edited by gussabina 2017-09-19 09:15:15)

  • Registered: 2017-07-02
  • Posts: 24

Re: [SOLVED] Decrypt problem in WolfSSL

David:

While reviewing the code, I realized I have a high priority task which looks for packets arriving from a Radio interface and forward them via MQTT Pub which utilizes wolfSSL_write() to send data to the cloud.
I also have another (lower prority) task which waits for data (MQTT Sub) from the cloud and utilizes wolfSSL_read() for this.
It seems at some point, while the latest task is checking for arrived packets from the cloud, it might be interrupted by the first one to send the radio packets to the cloud. I'm not sure at which point the task is being interrupted but it's possible that while using wolfSSL_read() this happen, leading to a potential problem.

How should I protect the wolfSSL access? Should I use a lock for avoiding concurrent access to wolfSSL_read() and  wolfSSL_write() functions?

UPDATE: it seems you wrote by the time I was writing this message... Thanks for clarifications. I just want to add I'm using MQTT Eclipse Paho as it was the one coming with the example. Anyway, I will continue working on following your advices.

Thank you for your support.
Regards;
Gus

Share

  • Registered: 2015-10-07
  • Posts: 415

Re: [SOLVED] Decrypt problem in WolfSSL

Hi Gussabina,

That explains the issue you are seeing. The protection needs to happen at the MQTT protocol level since MQTT does not provide a way to split up publish messages. There is a header and the message follows behind which may span over multiple TLS / TCP packets. Additionally the publish being sent can respond with a confirmation if using Qos of 1 or 2.

So the answer is you will need to implement a way to make sure each thread has exclusive access to the MQTT library. I'd recommend either have single thread and using FreeRTOS queue to pass between threads or adding mutex protection around both MQTT tasks and implement non-blocking MQTT mode so it does not block and each thread gets access to the socket.

Thanks,
David Garske, wolfSSL

Share

  • Registered: 2017-07-02
  • Posts: 24

Re: [SOLVED] Decrypt problem in WolfSSL

Hello David:

I was able to rework the code so all mqtt calls are on the same thread, and also increased stack and heap for FreeRTOS tasks to avoid possible memory corruption... But I still have the Decryption issue....

I was also able to narrow down the test case to check when exactly the issue occurs, and it seems to happen when more than one packet is received very fast (at least relative to what the receiver can process) by the internet interface (I tested with both ethernet with lwIP and WiFi with on-chip stack), which apparently cause one of the packets are dropped by the driver and then I get the decryption problem.. So my question is, when a TLS session is in course, does each packet encryption relates to the previous one? I mean, in case a packet is lost, will next packet be decrypted without problem? Since the packet is encrypted, I can't check the content to see if this is exactly what it was sent from the other end....

Thanks for your support.
Regards;
Gus

Share

9 Reply by gussabina (edited by gussabina 2017-09-21 19:37:02)

  • Registered: 2017-07-02
  • Posts: 24

Re: [SOLVED] Decrypt problem in WolfSSL

UPDATE:

After many hours of debugging, I think I found the problem...

When the interface receives more than one packet from the last time the code checked for new packets, both messages are passed as one (combining both), and it seems this is not valid for the decryption. I still see with wireshark that all messages are received separately, but the network interface passes them as one.  Can be this the case?

Socks 69
Rev Buffer size 1
Rev Buffer size 1
Rev Buffer size 38
Subscribed Topic = devControl/ABC106{
                                      "Msg":"Help!"
                                                   }

Message (size: 19) = {
                       "Msg":"Help!"
                                    }

Socks 138
Rev Buffer size 1
Rev Buffer size 1
Rev Buffer size 38
Subscribed Topic = devControl/ABC106{
                                      "Msg":"Help!"
                                                   }

Message (size: 19) = {
                       "Msg":"Help!"
                                    }

Socks 69
.....SendAlert.....2 20
----> Detecta problema Decrypting

If this is causing the error, I will see how can I modify it to avoid this problem.
Thanks
Gus

Post's attachments

error.PNG
error.PNG 53.98 kb, 1 downloads since 2017-09-22 

You don't have the permssions to download the attachments of this post.

Share

  • Registered: 2015-10-07
  • Posts: 415

Re: [SOLVED] Decrypt problem in WolfSSL

Hi Gussabina,

The IO read callback will ask for X bytes. If your network return X+10 bytes (for example) you will need to cache the extra 10 bytes until the next callback. Typically when you ask a network stack for X bytes it only returns X bytes. However if you are using something besides a network stack and asking the hardware for a buffer, you may receive more than X bytes. Its up to you to cache those extra bytes until the next IO read callback.

Let me know if that helps or not.

Thanks,
David Garske, wolfSSL

Share

  • Registered: 2017-07-02
  • Posts: 24

Re: [SOLVED] Decrypt problem in WolfSSL

Hello David:

Thank you!
This definitely helped me to solve the problem.

Thanks again.
Regards;
Gus

Share