This post was written by Guido Vranken and originally published on his blog (May 13, 2020).
Larry Stefonic of wolfSSL contacted me after he’d noticed my project for fuzzing cryptographic libraries called Cryptofuzz. We agreed that I would write a Cryptofuzz module for wolfSSL.
I activated the wolfSSL module for Cryptofuzz on Google’s OSS-Fuzz, where it has been running 24/7 since. So far, Cryptofuzz has found a total of 8 bugs in wolfCrypt.
Larry and Todd Ouska then asked me if I was interested in writing fuzzers for the broader wolfSSL library. I was commissioned for 80 hours of work.
I started by implementing harnesses for the TLS server and client. Both support 5 different flavors of TLS: TLS 1, 1.1, 1.2, 1.3 and DTLS.
wolfSSL allows you to install your own IO handlers. Once these are in place, each time wolfSSL wants to either read or write some data over the network, these custom handlers are invoked, instead calling recv() and send() directly.
For fuzzing, this is ideal, because fuzzers are best suited to operate on data buffers rather than network sockets. Working with actual sockets in fuzzers is possible, but this tends to be slower and more complex than piping data in and out of the target directly using buffers.
Hence, by using wolfSSL’s IO callbacks, all actual network activity is sidestepped, and the fuzzers can interact directly with the wolfSSL code.
Emulating the network
In the write callback, I embedded some code that specifically checks the outbound data for uninitialized memory. By writing this data to /dev/null, it can be evaluated by valgrind and MemorySanitizer.
Furthermore, I ensured that my IO overloads mimic the behavior of a real network.
On a real network, a connection can be closed unexpectedly, either due to a transmission failure, a man-in-the-middle intervention or as a deliberate hangup by the peer.
It is interesting to explore the library’s behavior in the face of connection issues, as this can activate alternative code paths that normally are not traversed, so this strategy harbors the potential to find bugs that are missed otherwise.
For example, what if wolfSSL wants to read 50 bytes from a socket, but the remote peer sends only 20?
These are situations that are feasible if an attacker were to deliberately impose transfer throttling in their communication with an endpoint running wolfSSL.
Addition and subtraction have shown to pose a challenge in programming, especially when they pertain to array sizes; many buffer overflows and infinite loops in software (not wolfSSL in particular) can be traced back to off-by-one calculations and integer overflows.
Networking software like wolfSSL needs to keep a tally of completed and pending transmissions and in light of this it is a worthwhile experiment to observe what will happen when faced with uncommon socket behavior.
Finding instances of resource exhaustion
Buffer overflows are not the only kind of bug software can suffer from.
For example, it would be unfortunate if an attacker could bring down a TLS server by sending a small, crafted packet.
Fuzzing can be helpful in finding denial of service bugs. Normally, fuzzers use code coverage as a feedback signal. By instead using the branch count or the peak memory usage as a signal, the fuzzer will tend to find slow inputs (many branches taken means a long execution time) or inputs that consume a lot of memory, respectively.
Several years ago I implemented some modifications to libFuzzer which allow me to easily implement fuzzers that find denial-of-service bugs. For my engagement with wolfSSL, I applied these techniques to each fuzzer that I wrote. I ended up providing three binaries per fuzzer:
- a generic one that seeks to find memory bugs, using code coverage as a signal
- one that tries to find slow inputs by using the branch count as a signal
- one that finds inputs resulting in excessive heap allocation
Emulating allocation failures
Using wolfSSL_SetAllocators(), wolfSSL allows you to replace its default allocation functions. This opens up interesting possibilities for finding certain bugs.
One thing I did in my custom allocator was to return an invalid pointer for a malloc() or realloc() call requesting 0 bytes. This way, if wolfSSL would try to dereference this pointer, a segmentation fault will occur.
This special code is needed because even AddressSanitizer will not detect access to a 0-byte allocated region, but it is important to test for, as such behavior can lead to real crashes on systems like OpenBSD, which intentionally return an invalid pointer from malloc(0), just like my code does.
Another possibility of implementing your own memory allocator is that it can be designed to fail sometimes.
On most desktop systems, malloc() always succeeds, but that may not be the case universally, especially not on resource-constrained systems which cannot resort to page swapping for acquiring additional memory.
Allocation failures activate code paths which are normally not accounted for by unit tests. I implemented this behavior for all fuzzers I wrote for wolfSSL.
In the TLS-specific code, 5 bugs were found.
Fuzzing auxiliary code
TLS is large and complex, and it can take fuzzers a while to traverse all its code paths, so in the interest of efficiency, I wrote several additional fuzzers specifically aimed at subsets of the library, like X509 certificate parsing (historically a wellspring of bugs across implementations), OCSP request and response handling (for which a subset of HTTP is implemented) and utility functions like base64 and base16 coders.
This approach found 9 additional bugs.
Testing the bignum library
wolfSSL comes with a bignum library that it uses for asymmetric cryptography. Because it is imperative that computations with bignums are sound, I took a project of mine called bignum-fuzzer (which has also found security bugs in other bignum libraries, like OpenSSL’s CVE-2019-1551) and appropriated it for use with wolfSSL. It is not only able to find memory bugs, but also incorrect calculation results.
I set out to test the following sub-libraries in wolfSSL:
- Normal math
- Single-precision math (–enable-sp)
- Fastmath (–enable-fastmath)
5 instances of incorrect calculations were found. The other bugs involved invalid memory access and hangs.
wolfSSH
In addition to wolfSSL and wolfCrypt, I also spent some time looking at wolfSSH, which is the company’s SSH library offering.
In this component I uncovered 7 memory bugs, 1 memory leak and 1 crash bug.