1 (edited by carlo 2020-07-04 15:13:57)

Topic: Mixed reading and writing from wolfSSL_connect is giving me a headache

After literally weeks of work, I finally managed to integrate wolfssl into libevio.
But it was really painful to deal with the duality of wolfssl_connect
which, even when called when it says it WANT_WRITE - happily continuous
to read after writing.

Here is some of the "code" that I wrote (more comment than code, but
otherwise nobody would understand what is going on wink...

  // Bits of TLS::m_session_state
  //  _ post_handshake
  // / _ want_write
  // |/  _ inside_do_handshake
  // || /
  // || |   Required action   Possible transitions to  Condition result
  // || |
  // 00x0    stop              00x0, 00x1, 01x0, 10x0   WasTrue
  // 00x1    stop              00x0, 01x0, 10x0         WasTrue
  // 01x0    do not stop       01x0                     False
  // 01x1    stop              00x0, 01x0, 10x0         WasTrue
  // 10x0    stop if           10x0                     obuffer->StreamBufConsumer::nothing_to_get() (False or WasTrue)
  // 10x1   not possible
  // 11x0   not possbile
  // 11x1   not possible
  //
  // If the post_handshake bit is not set, then it was never set before.
  // Therefore, since in the immediate past is_blocked_or_handshake_finished()
  // returned true, the inside_do_handshake bit must have been set.
  // Hence, if we see that neither post_handshake nor inside_do_handshake
  // are set then the read thread returned from do_handshake and
  // signalled if it wants to continue with reading or writing by clearing
  // or setting the want_write bit; however, since we also get here when
  // state == 0, it is possible that in that case it was THIS thread that
  // just reset the inside_do_handshake bit (after having just executed
  // do_handshake). In that case the read thread is still running and
  // could cause transitions from 00x0 to any other state, but that doesn't
  // change the fact that the required action to stop is a WasTrue.
  //
  // Hence, if we see that handshake_wants_write_and_not_blocked(state)
  // then the read thread must have left do_handshake (resetting
  // inside_do_handshake) and set the want_write bit. That means the
  // handshake is not finished yet and the handshake wants to continue
  // with writing (not reading!).
  // Therefore the *read* thread will not re-enter do_handshake and
  // thus m_session_state can't change anymore.
  //
  // Therefore this condition must return fuzzy::WasTrue for all states
  // except where handshake_wants_write(state), in which case it should
  // return fuzzy::False.
  //
  // If is_post_handshake(state) then instead it should return
  // obuffer->StreamBufConsumer::nothing_to_get(). Note that a state of
  // post_handshake can also no longer change, leaving the fuzzy::False
  // returned by StreamBufConsumer::nothing_to_get() at fuzzy::False.
  utils::FuzzyBool must_stop_output_device(OutputBuffer const* obuffer)
  {
    int state = m_session_state.load(std::memory_order_relaxed);
    if (handshake_wants_write_and_not_blocked(state))
      return fuzzy::False;
    utils::FuzzyBool output_buffer_is_empty = obuffer->StreamBufConsumer::nothing_to_get();
    if (output_buffer_is_empty.is_false())
      m_session_state.fetch_or(s_have_plain_text, std::memory_order_release);
    if (is_post_handshake(state))
      return output_buffer_is_empty;
    return fuzzy::WasTrue;
  }

  // Bits of TLS::m_session_state
  //  _ post_handshake
  // / _ want_write
  // |/  _ inside_do_handshake
  // || /
  // || |   Required action   Possible transitions to  Condition result
  // || |
  // 00x0    do not stop       00x0                     False
  // 00x1    stop              00x0, 01x0, 10x0         WasTrue
  // 01x0    do not stop       00x0, 01x0, 01x1, 10x0   False
  // 01x1    do not stop       00x0, 01x0, 10x0         False
  // 10x0    do not stop       10x0                     False
  // 10x1   not possible
  // 11x0   not possbile
  // 11x1   not possible
  //
  // We can not stop the input device when the state is 01x0 because
  // that means that the write thread is running which might stop
  // the output device (and stopping both input and output device
  // could terminate the application; or more specifically it means
  // that 'we are done' with this device in evio terms - which is
  // obviously not true).
  //
  // Because that state (01x0) can transition to 01x1 (when the
  // write thread enters do_handshake()), that state can not cause
  // us to stop the input device either or the result of state 01x0
  // would need WasFalse to be returned - which is not allowed (or
  // rather, that can't work: due to a race condition we could
  // *miss* stopping on state 01x1).
  //
  // Obviously we do not want to stop the input device when the
  // state is 00x0 or 10x0; in the first case because the handshake
  // is not finished and it wants to read, and in the second because
  // the handshake is finished and we can return to the sane strategy
  // of never stopping to read an input device until we're done with it.
  //
  // We HAVE to stop when the state is 00x1 because reading is
  // required but the write thread is handling the handshake at
  // the moment (so we can't). Not stopping could lead to an immediate
  // return to read_from_fd and thus cause a tight loop using 100% cpu.
  //
  // None of the other states can transition to 00x1, so that we
  // can return fuzzy::False for all of them as is required (see above).
  //
  // The reason that a transition to 00x1 is not possible is because
  // 1) when the write thread is inside do_handshake (xxx1) the only
  //    possible transition is when the write thread leaves do_handshake
  //    which always resets the least significant bit (xxx1 --> xxx0).
  // 2) Once the state is 10x0 the handshake is finished and the state
  //    won't return to an unfinished handshake.
  // 3) If the write thread is outside do_handshake and the state is
  //    01x0 then the only possible (first) transition is to 01x1 (
  //    which subsequently could change back to 01x1, to 10x0 or to
  //    00x0, and)
  // 4) If the write thread is outside do_handshake and the state is
  //    00x0 then the write thread is stopped, so it won't make any
  //    additional changes anymore.
  //
  // Not stopping in the state 01x1 is unfortunate, but discussed
  // elsewhere.
  utils::FuzzyBool must_stop_input_device() const
  {
    int state = m_session_state.load(std::memory_order_relaxed);
    return handshake_wants_read_and_blocked(state) ? fuzzy::WasTrue : fuzzy::False;
  }

and....

      // Did the handshake finish successfully?
      if (TLS::handshake_completed(state))
      {
        Dout(dc::tls, "Handshake completed!");
        m_connected_flags |= is_connected;
        // Do the m_connected() callback at this point  (as opposed to when the TCP connection was established),
        // as in most cases it will be used as a "you can now send/receive data" signal...
        if (m_connected)
        {
          int count = allow_deletion_count;
          m_connected(allow_deletion_count, true);
          if (allow_deletion_count > count)
            // Device is marked for deletion.
            return;
        }
        // It is impossible to test if the output buffer is empty from this thread.
        //
        // It would work to simply start the output device and let the write thread deal with it (that is,
        // the write thread would stop the output device again if the buffer turns out to be empty).
        // However, if there is a way to avoid a needless start and subsequent stop then that would be preferable.
        //
        // If we do some fuzzy test - and based on that start the output device, then
        // nothing is lost. The only thing that we want to avoid is that we end up with
        // a stopped output device while there is something in the output buffer.
        // It is not possible however that by doing nothing we end in that state unless
        // there is already something in the output buffer (that was flushed) and no
        // new flush happens after this point (from another thread).
        //
        // When something is, or was, written to the output buffer and flushed - then that
        // caused the output device to be started. So, it is necessary that subsequently 
        // this was ignored from write_to_fd() because the TLS handshake had not finished
        // yet.
        //
        // Moreover, the output device begins started, so it must have been stopped in
        // the meantime (as part of the TLS handshake), which happens exclusively from
        // the write thread.
        //
        // Therefore, it is possible to know if there is something in the (plain text)
        // output buffer as detected by the write thread when it stopped the output
        // device.
        if (m_tls.need_start_output_device(state))   
          start_output_device(state_t::wat(m_state));
        // It is very unlikely that there is more to read, immediately after the handshake.
        break;
      }

      // Stopping the input device could cause the application to exit if
      // this is the only device and the output device is stopped too.
      // Therefore, we will only stop the input device if
      // 1) the handshake is not finished,
      // 2) the handshake wants to read,
      // 3) the write thread is inside do_handshake.
      // See TLS::must_stop_input_device for the detailed argumentation.
      utils::FuzzyCondition condition_must_stop_input_device([this]{
          return m_tls.must_stop_input_device();
      });
      if (condition_must_stop_input_device.is_momentary_false())
      {
        Dout(dc::tls, "Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = " << state << ").");
        continue;
      }

I suspect nobody actually read all that, but you get the idea.

Share

Re: Mixed reading and writing from wolfSSL_connect is giving me a headache

The above code (including reasoning) then leads to the following:

ThreadPool16    TLS           :   Entering TLS::do_handshake()
ThreadPool16    TLS           :     wolfSSL_connect((WOLFSSL*)0x7f023c0018d0) = <unfinished>
ThreadPool16    EVIO          :         Entering TLS::send(0x7f023c00c280, 320) [(WOLFSSL*)0x7f023c0018d0, FD:0x560b46510c80]
ThreadPool16    SYSTEM        :           send(15, "\x16\x03\x03\x01;\x01\x00\x017\x03\x03&\xD2(Z\xB0\xC8\xF0LP\x92r:\x10\n=\xFCH" \xF4!?\xDC0\xE6\x9F\x1F\xA1Q\xD5{\xD3\x00\x00X\x13\x01\x13\x02\x13\x03\x13\x04\x13\x05\xC0\xB4\xC0\xB5\xC0,\xC0+\xC00\xC0/\x00\x9F\x00\x9E\xCC\xA9\xCC\xA8\xCC\xAA\xC0'\xC0#\xC0(\xC0$\xC0\n\xC0\t\xC0\a\xC0\b\xC0\x14\xC0\x13\xC0\x11\xC0\x12\xC0\xAC\xC0\xAE\xC0\xAF\x00k\x00g\x009\x003\x00\x16\xCC\x14\xCC\x13\xCC\x15\xC0\x06\x00E\x00\x88\x00\xBE\x00\xC4\x01\x00\x00\xB6\x003\x00G\x00E\x00\x17\x00A\x04*\xB6!\xD4//\xCBk\xD4\xDA\xE0+#T,*\xEAN\xEDF\xA4\xED5\xE8\xDE\xA7\xE7\xC8\\\x84\x14#\xB1j\x7Fl\x11H\xE5\xE1\x03\x80\xC1\a\x1CBj\x0E\xBD\xC3\xA7\xFB\x12\x99\xDD2b\x17\x92,rb\x0F|\x00+\x00\t\b\x03\x04\x03\x03\x03\x02\x03\x01\x00\r\x00"\x00 \x06\x03\x05\x03\x04\x03\x02\x03\b\a\b\x06\b\v\b\x05\b\n\b\x04\b\t\x06\x01\x05\x01\x04\x01\x03\x01\x02\x01\x00\v\x00\x02\x01\x00\x00\n\x00&\x00$\x00\x19\x00\x1C\x00\x18\x00\e\x00\x1E\x00\x17\x00\x16\x00\x1A\x00\x1D\x00\x15\x00\x14\x00\x13\x00\x12\x00\x10\x00\x11\x00\x0F\x01\x01\x01\x00\x00\x16\x00\x00\x00\x17\x00\x00", 320, 0) = 320
ThreadPool16    EVIO          :         Entering TLS::recv(0x7f023c001b98, 5) [(WOLFSSL*)0x7f023c0018d0, FD:0x560b46510c80]
ThreadPool16    SYSTEM        :           recv(15, {"\x16\x03\x03\x00Z"}, 5, 0) = 5
ThreadPool16    EVIO          :         Entering TLS::recv(0x7f023c00cbb0, 90) [(WOLFSSL*)0x7f023c0018d0, FD:0x560b46510c80]
EventLoopThr    SYSTEM        : <continued> 1
ThreadPool16    SYSTEM        :           recv(15, {"\x02\x00\x00V\x03\x03\x83\xA3\xBCa"\xA1.\xB7\xAC\b\xC9N, \xAC\x1D\xAFk\x8E\xAB\x9Ep\x01\xEBa\xFC\x9AQ~>\xE8\xE9 \\\x7F\b\xED%#9\xB0P\xCE)]\x10\xE6X\xFE\xA8n\x1F\xA1\xC4~\x8B7?\xB3\xFC9)\x9F\xD1\xF8\xC00\x00\x00\x0E\x00\x16\x00\x00\x00\v\x00\x02\x01\x00\x00\x17\x00\x00"}, 90, 0) = 90
EventLoopThr    EVIO          : epoll_pwait event(s) EPOLLOUT of fd FD:0x560b46510c80 ignored because the event(s) EPOLLOUT is/are already being handled by the thread pool.
EventLoopThr    SYSTEM        : epoll_pwait() = <unfinished>
ThreadPool16    EVIO          :         Entering TLS::recv(0x7f023c001b98, 5) [(WOLFSSL*)0x7f023c0018d0, FD:0x560b46510c80]
EventLoopThr    SYSTEM        :     FD:0x560b46510c80: 15, FDS_RW|FDS_R_OPEN|FDS_W_OPEN|FDS_SAME|FDS_R_ADDED|FDS_W_ADDED|FDS_R_ACTIVE|FDS_W_ACTIVE
ThreadPool16    SYSTEM        :           recv(15, {"\x16\x03\x03\x04\xAC"}, 5, 0) = 5
EventLoopThr    SYSTEM        : <continued> 1
EventLoopThr    EVIO          : epoll_pwait event(s) EPOLLOUT of fd FD:0x560b46510c80 ignored because the event(s) EPOLLOUT is/are already being handled by the thread pool.
ThreadPool16    EVIO          :         Entering TLS::recv(0x7f023c00cc20, 1196) [(WOLFSSL*)0x7f023c0018d0, FD:0x560b46510c80]
EventLoopThr    EVIO          : epoll_pwait new event(s): EPOLLIN
EventLoopThr    EVIO          : Queuing I/O event EPOLLIN for FD:0x560b46510c80 in thread pool queue #0
EventLoopThr    SYSTEM        : epoll_pwait() = <unfinished>
EventLoopThr    SYSTEM        :     FD:0x560b46510c80: 15, FDS_RW|FDS_R_OPEN|FDS_W_OPEN|FDS_SAME|FDS_R_ADDED|FDS_W_ADDED|FDS_R_ACTIVE|FDS_W_ACTIVE
ThreadPool10    THREADPOOL    : Beginning of thread pool main loop (q = #0)
ThreadPool10    THREADPOOL    : Not going idle.
ThreadPool10    EVIO          : Beginning of handling event EPOLLIN for FD:0x560b46510c80.
ThreadPool10    EVIO          : Entering TLSSocket::read_from_fd({1}, 15) [FD:0x560b46510c80]
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool16    SYSTEM        :           recv(15, {"\v\x00\x04\xA8\x00\x04\xA5\x00\x04\xA20\x82\x04\x9E0\x82\x03\x86\xA0\x03\x02\x01\x02\x02\x01\x010\r\x06\t*\x86H\x86\xF7\r\x01\x01\v\x05\x000\x81\x941\v0\t\x06\x03U\x04\x06\x13\x02US1\x100\x0E\x06\x03U\x04\b\f\aMontana1\x100\x0E\x06\x03U\x04\a\f\aBozeman1\x110\x0F\x06\x03U\x04\n\f\bSawtooth1\x130\x11\x06\x03U\x04\v\f\nConsulting1\x180\x16\x06\x03U\x04\x03\f\x0Fwww.wolfssl.com1\x1F0\x1D\x06\t*\x86H\x86\xF7\r\x01\t\x01\x16\x10info@wolfssl.com0\x1E\x17\r180413152310Z\x17\r210107152310Z0\x81\x901\v0\t\x06\x03U\x04\x06\x13\x02US1\x100\x0E\x06\x03U\x04\b\f\aMontana1\x100\x0E\x06\x03U\x04\a\f\aBozeman1\x100\x0E\x06\x03U\x04\n\f\awolfSSL1\x100\x0E\x06\x03U\x04\v\f\aSupport1\x180\x16\x06\x03U\x04\x03\f\x0Fwww.wolfssl.com1\x1F0\x1D\x06\t*\x86H\x86\xF7\r\x01\t\x01\x16\x10info@wolfssl.com0\x82\x01"0\r\x06\t*\x86H\x86\xF7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0F\x000\x82\x01\n\x02\x82\x01\x01\x00\xC0\x95\b\xE1WA\xF2qm\xB7\xD2EA'\x01e\xC6E\xAE\xF2\xBC$0\xB8\x95\xCE/N\xD6\xF6\x1C\x88\xBC|\x9F\xFB\xA8g\x7F\xFE\\\x9CQu\xF7\x8A\xCA\a\xE75/\x8F\xE1\xBD{\xC0/|\xABd\xA8\x17\xFC\xCA]{\xBA\xE0!\xE5r.o.\x86\xD8\x95s\xDA\xAC\eS\xB9_?\xD7\x19\r%O\xE1ccQ\x8B\vd?\xADC\xB8\xA5\x1C\\4\xB3\xAE\x00\xA0c\xC5\xF6\x7F\vYhxs\xA6\x8C\x18\xA9\x02m\xAF\xC3\x19\x01.\xB8\x10\xE3\xC6\xCC@\xB4i\xA3F3i\x87n\xC4\xBB\x17\xA6\xF3\xE8\xDD\xADs\xBC{/!\xB5\xFDfQ\f\xBDT\xB3\xE1m_\x1C\xBC#s\xD1\t\x03\x89\x14\xD2\x10\xB9d\xC3*\xD0\xA1\x96J\xBC\xE1\xD4\x1A[\xC7\xA0\xC0\xC1cx\x0FD702\x96\x802#\x95\xA1w\xBA\x13\xD2\x97s\xE2]%\xC9j\r\xC39`\xA4\xB4\xB0iBB\t\xE9\xD8\b\xBC3 \xB3X"\xA7\xAA\xEB\xC4\xE1\xE6a\x83\xC5\xD2\x96\xDF\xD9\xD0O\xAD\xD7\x02\x03\x01\x00\x01\xA3\x81\xFC0\x81\xF90\x1D\x06\x03U\x1D\x0E\x04\x16\x04\x14\xB3\x112\xC9\x92\x98\x84\xE2\xC9\xF8\xD0;n\x03B\xCA\x1F\x0E\x8E<0\x81\xC9\x06\x03U\x1D#\x04\x81\xC10\x81\xBE\x80\x14'\x8Eg\x11t\xC3&\x1D?\xED3c\xB3\xA4\xD8\x1D0\xE5\xE8\xD5\xA1\x81\x9A\xA4\x81\x970\x81\x941\v0\t\x06\x03U\x04\x06\x13\x02US1\x100\x0E\x06\x03U\x04\b\f\aMontana1\x100\x0E\x06\x03U\x04\a\f\aBozeman1\x110\x0F\x06\x03U\x04\n\f\bSawtooth1\x130\x11\x06\x03U\x04\v\f\nConsulting1\x180\x16\x06\x03U\x04\x03\f\x0Fwww.wolfssl.com1\x1F0\x1D\x06\t*\x86H\x86\xF7\r\x01\t\x01\x16\x10info@wolfssl.com\x82\t\x00\x86\xFF\xF5\x8E\x10\xDE\xB8\xFB0\f\x06\x03U\x1D\x13\x04\x050\x03\x01\x01\xFF0\r\x06\t*\x86H\x86\xF7\r\x01\x01\v\x05\x00\x03\x82\x01\x01\x00\xB4T`\xAD\xA0\x032\xDE\x02\x7F!J\x81\xC6\xED\xCD\xCD\xD8\x12\x8A\xC0\xBA\x82[u\xADT\xE3|\x80j\xAC.l N\xBEM\x82\xA7G\x13\\\xF4\xC6j+\x10\x99X\xDE\xABk|"\x05\xC1\x83\x9D\xCB\xFF<\xE4-Wj\xA6\x96\xDF\xD3\xC1h\xE3\xD2\xC6\x83K\x97\xE2\xC62\x0E\xBE\xC4\x03\xB9\a\x8A[\xB8\x84\xBA\xC59?\x1CX\xA7U\xD7\xF0\x9B\xE8\xD2E\xB9\xE3\x83.\xEE\xB6qV\xB9:\xEE?'\xD8w\xE8\xFBDHe'GL\xFB\xFEr\xC3\xAC\x05{\x1D\xCB\xEB^e\x9A\xAB\x02\xE4\x88[;\x8B\v\xC7\xCC\xA9\xA6\x8B\xE1\x87\xB0\x19\x1A\f(Xo\x99R~\xED\xB0:h;\x8C\n\btr\xAB\xB9\t\xC5\xED\x04~o\v\x1C\t!\xD0\xCD\x7F\xF9\xC4^' \xE4\x85sR\x05\xD2\xBA\xF8\xD5\x8FA\xCC#.\x12m\xBC1\x98\xE7c\xA3\x8E&\xCD\xE8+\x88\xEE\xE2\xFE:tR4\x0E\xFD\x12\xE5^iP 14\xE41\xF1\xE7\xE4[\x03\x13\xDA\xACAl\xE7\xCF+"}, 1196, 0) = 1196
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool16    EVIO          :         Entering TLS::recv(0x7f023c001b98, 5) [(WOLFSSL*)0x7f023c0018d0, FD:0x560b46510c80]
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool16    SYSTEM        :           recv(15, 0x7f023c001b98, 5, 0) = -1: EAGAIN (Resource temporarily unavailable)
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool10    TLS           :   Trying buffer again because condition_must_stop_input_device.is_momentary_false() returned true (state = 5).
ThreadPool16    TLS           :     <continued> -1 (WOLFSSL_ERROR_WANT_READ: non-blocking socket wants data to be read)
ThreadPool10    TLS           :   Entering TLS::do_handshake()
ThreadPool16    TLS           :     wolfSSL_want_read((WOLFSSL*)0x7f023c0018d0) returned true.

In other words: waste of CPU time...

This is basically the result of the write thread starting the handshake (writing to the fd), causing
data being sent by the server, causing the fd becoming readable, WITHOUT that this is communicated
by a (handshake) WANT_READ: because wolfSSL_connect doesn't return.

Subsequently my read thread sees that there is something to read on the fd, but is not allowed to
call wolfSSL_connect (because the other thread is inside it) and neither can stop monitoring the fd
for readability (as explained in the comments).

Share

Re: Mixed reading and writing from wolfSSL_connect is giving me a headache

Carlo,

Thank you for your notes. As I mentioned in my reply on https://www.wolfssl.com/forums/topic156 … shake.html One can not manage read/writes during the handshake. The internal state machine will manage it based on the RFC specs. Only once the handshake has completed and the connection is established can the end-application then take over and manage reads/writes separately.

Warm Regards,

K