wolfSSL Quality Assurance

The wolfSSL ecosystem consists of several software modules and components, each with specific goals and purposes. We make sure all our software products are engineered using the quality standards required by our process.

Each step in the software lifetime is regulated by strict rules and testing criteria (including stringent fuzz based testing) that ensure the detection of defects and regressions in the code very early.

Quality assurance

The first verification for the functionality of the code is performed locally, on the development PC of the contributor. Git commit and push hooks ensure that the code can be submitted only if it passes a first set of functionality and unit tests. Once the pull request is published, a full round of non-regression and integration tests are automatically started and the status of the pull request is updated with the test results. In order for the pull request to be accepted for inclusion, it must pass the peer review and the non-regression and integration tests. The tests are automatically retriggered during the review process every time that the code for the pull request is modified.

Quality control automation

At wolfSSL we have deployed an hybrid (on-site + cloud based) infrastructure, using Jenkins to coordinate the workload between the nodes to apply quality control on a regular basis. This includes the execution of software tests every time a contribution is evaluated for inclusion in the mainline, as well as other types of quality control applied on a regular basis (e.g. nightly, weekly).

The reason behind the hybrid approach is due to the portability trait of the wolfSSL software ecosystem. The software must run on several different hardware architectures, and interoperate with specific hardware components, such as hardware crypto modules and TPMs. Using physical machines on some of the jenkins nodes provides mechanisms to configure and control specific hardware targets, including microcontroller boards that can be configured and programmed automatically.

Some tests require a long time to run. Continuous integration tools are very useful to split the application of the quality control jobs over a longer time, to ensure that every test is performed on a regular basis.

Formal algorithms and modules verification

In order to validate the correctness and the adherence to the standards implemented, wolfSSL software components are tested using tools and procedures recommended by NIST. This includes a full set of functional tests using a set of well-known input values (test vectors) and expected results. The correctness of many cryptographic algorithms can also be verified by inspecting the intermediate results of the calculations. 

NIST also issues a series of publications (FIPS 140) coordinating the requirements and standards for cryptographic modules for use by departments and agencies of the federal Government. Through the effort of specialized accredited third-party laboratories across the U.S. and Canada, two validation programs are made available to certify the compliance with the FIPS 140 regulations. wolfCrypt has achieved the FIPS 140-2 certification, and has already applied for the recently approved FIPS 140-3 certificate. FIPS certification requires that the cryptographic module is successfully submitted through the validation of the two programs:

• Cryptographic Algorithm Validation Program (CAVP) provides validation testing of approved cryptographic algorithms implemented.
• Cryptographic Module Validation Program (CMVP) certifies the module for use by the Government and regulated industries for securing sensitive data and information. 

The same types of tests used in CAVP/CMVP are repeated on the mainline automatically, to ensure that modifications in the code do not impact on the integrity of the algorithms and the modules as specified by FIPS 140.

Interoperability tests

One very effective way of verifying that the behavior of the protocols remains the same throughout the continuous integration, is by running interoperability tests with different implementations of the same standards. WolfSSL quality control infrastructure provides a number of scheduled tests that use a different implementation as the remote endpoint of the communication, and to compare results of cryptography operations starting from common vectors.

Unit tests

Unit tests are mandatory for all core modules, and run on the developer’s machine upon new commits.

The coverage of the unit tests is measured on a weekly basis, to ensure that there are no coverage regressions when adding new functionality to the code. WolfSSL developers receive a full code coverage report in their mailboxes every week thanks to the automation provided by the Jenkins infrastructure for continuous integration.

API consistency verification

One specific set of tests verifies that changes in the implementation do not alter the usage of the API from the application development perspective. These tests are never updated, with the only exception of adding new functions to the API. The API must remain consistent across versions, as it is the contract between the application and the library. Verifying all these aspects is the goal of the subset of tests that are running nightly to verify the adherence to the requirements.

Integration tests

Due to wolfSSL portability, it is necessary to expand the test domain by including custom configurations, which require to compile the software for different architectures and combination of compile-time configuration options and test applications. Both real and virtual machines are used as targets for running the test suite. Automating the tests on different architectures (x86, ARM, PowerPC, RISC-V, MIPS, …) ensures that architecture-specific regressions or bugs can be detected and identified early during the process, and the expected behavior remains consistent in all cases. Thanks to the hybrid model of our continuous integration infrastructure, several targets are connected to Jenkins nodes that are in charge of running the software tests through a wide range of specific hardware and software configurations and use cases.

Safety assessment: looking under the hood

The continuous integration infrastructure also automates the execution of several analysis tools. 

Static analysis tools look for any inconsistency in the code exploring all the different combinations of compile-time options, and following different code paths. These tools can detect a wide range of programming mistakes, potential errors, and undefined behaviors in the language that may not be covered by the compiler, by applying rigorous checks at the source code level. The tools used by wolfSSL and automated in the CI include cppcheck, clang static analyzer (scan-build), Facebook infer and others.

Memory analysis is performed on a regular basis to look for bugs related to memory handling. WolfSSL uses valgrind memcheck tool, clang sanitizer and other dynamic analysis to run the code. These tools detect memory errors such as accessing uninitialized or previously freed memory, using undefined or uninitialized values, memory leaks and more.

Fuzzers are a very important resource to improve the robustness of the code towards unexpected situations. The goal of these tools is to attempt to cause malfunctions in the code by injecting a large number of random inputs in quick succession. Fuzzing is often a very effective way to detect bugs and vulnerabilities in the code that could go unnoticed for a long time. At wolfSSL we constantly run fuzzers to feed the API functions and the transport back-end, periodically rotating all the possible seed values for the PRNG regulating the output values mutation.  With mutation-fuzzing, a bug that is triggered with a given seed value can be reproduced by relaunching the same test with the same seed value manually, allowing for easy reproducibility and analysis through instruments and debuggers. Since these kind of tools must be aware of the application domain, the protocol structures and the characteristics of the data, wolfSSL uses two main fuzzers that have been written for the purpose. WolfFuzz operates over memory buffers and fuzzes the internal cryptography operations. This mechanism allows very fast fuzzing and the entire range of 4 trillion PRNG seeds is tested in three months. A second tool, wolfSSL Network Fuzzer, runs over TCP/IP. For this reason it is much slower but more flexible to test the code that enables security for data in motion.

Vulnerability management

At wolfSSL we take vulnerabilities very seriously, and we are committed to release a new version of the software within 36h from the disclosure. This ensures that, in case of responsible disclosure, the vulnerability is fixed way before its details or any proof of concept to reproduce are made public.

A vulnerability claim triggers emergency procedures, consisting in a Standard Operating Procedure (S.O.P.)  that has to be completed to speed up the resolution of the issue and the release of a new version. The vulnerability claim is verified within the first 120 minutes. In this phase, a document is created to be distributed internally to the engineering team, to document the issue and all available instruments along with instructions on how to reproduce it so that the error can be confirmed and the proposed fix can also be assessed to determine if the issue has been addressed completely. The fix may take between a few minutes up to 24h, depending on the complexity of the issue. When the fix is ready it is submitted either in the form of a internal patch or a public pull request when it has been determined that the fix will not leak critical information to would-be attackers that monitor the wolfSSL git repository. The path or public PR is then reviewed by multiple engineers, since code changes are still validated through both manual code reviews and the test procedures. After a few iterations in the review process loop, the automated integration server verifies the fix by running all the necessary pre-release tests. At the end of the verifications, if all the test passed, a new release is issued,  and all users and customers are notified through all available communication channels.