Here at wolfSSL we recently added fuzz testing to our testing processes. Security of CyaSSL software is always on our mind. As the software is used to secure connections and provide software security, the testing of how robust and secure CyaSSL is holds high importance.
As with the halting problem we know it is impossible to test every single possible path through the software but we practice an approach that is focused on lowering risk of failure. In addition to extensive automated tests we also make sure that we specifically test well known use cases. This post outlines some of our testing processes.
1. Build options: The first approach we use is by testing combinations of build options. Although there are too many potential combinations to test them all, this approach tests for potential issues with build option compatibility and also allows us to place a confidence value on build option combinations not yet used. We commonly use valgrind and scan-build when testing individual build options.
2. API testing: In each particular build, we test every available call to cover API uses.
3. Connection testing and data passing variables: To test this we start with simple connections and data, then as the test progresses we gradually increase the complexity of connection details.
4. Interop: We test for interoperability with the other open source TLS implementations, including OpenSSL and GnuTLS.
5. We then test interoperability outside of a closed environment and connect to unknown servers in the real world.
6. We build with a series of `real` applications, like cURL, wget, pppd, etc. For some of our customers with top level support, we build the new release with their application.
7. We test using a fuzzing software technique. This bombards the program with invalid, unexpected, and random data that then allows for observing if there is potential memory leaks or logic errors.
8. We engage in another ever expanding universe of benchmark testing, where we look at sizing, transmission rates, connection speeds, etc.
Much of our effort is automated by Jenkins (hat tip to that project!). Thanks for listening. If you have specific questions about how we test, please contact us at facts@wolfssl.com.