We are often asked about how we test wolfSSL. At this point, we believe we have testing that is quite robust, but we acknowledge that there is no such thing as perfect testing. With that knowledge in mind, we have the goal of incrementally improving and automating our testing rigs over time. Our overriding goal in testing is to have the most advanced and robust and ever improving testing that we can, subject to our resources.
We used to think of our testing plans in terms of a hierarchy, but we`ve improved on that thinking over time. We currently represent our testing as a series of concentric universes, with each successively larger universe being vastly larger and more complex than the smaller one inside of it. We`ll post some pictures in upcoming blog posts. For now, here`s a rough representation of our testing universes in order of sequence:
1. Build options: We have an extremely large number of build option combinations. So our most basic test universe is to build successfully with every possible combination of options.
2. API testing: We test every available call in a particular build.
3. Connection testing and data passing variables: We start with simple connection tests with limited data transmitted and then gradually dial up complexity.
4. Interop: We test for interoperability with the other open source TLS implementations, including OpenSSL and GnuTLS.
5. We then connect to unknown servers in the real world.
6. We then build with a series of `real` applications, like cURL, wget, pppd, etc. For some of our customers with top level support, we build the new release with their application.
7. Finally, we engage in another ever expanding universe of benchmark testing, where we look at sizing, transmission rates, connection speeds, etc. More to come on that topic, as it is quite popular!
Much of our effort is automated by Jenkins (hat tip to that project!). Thanks for listening. If you have specific questions about how we test, please contact us at facts@wolfssl.com.