Fuzzing is a well-known software testing technique, especially useful for testing smart contracts. Traditionally, as fuzzing, testers understood mainly Black Box Fuzzing or, more advanced, Property-Based Fuzzing. However, a new and innovative approach, Manually Guided Fuzzing, offers enhanced efficiency and effectiveness, filling the gaps left by traditional fuzzing techniques. This article will explore how Guided Fuzzing differs from Black Box Fuzzing, Property-Based Fuzzing, Differential Fuzzing and other techniques.
Understanding the Basics: Stateful vs. Stateless and Black Box vs. Grey Box vs. White Box Fuzzing
Fuzzing techniques can be classified based on their statefulness and the level of guidance provided:
Stateful vs. Stateless Fuzzing:
Stateless Fuzzing: Each test case is independent of the others in stateless fuzzing. The fuzzer does not maintain any knowledge of previous states or inputs, which limits its ability to discover bugs related to the sequence of operations.
Stateful Fuzzing: This method considers the system’s state, meaning the outcome of one test case can influence the subsequent ones. Stateful fuzzing is more effective for testing complex systems like smart contracts, where state transitions are critical. See the difference on this code snippet written in Wake Framework.
Black Box vs. Grey Box vs. White Box:
Black Box Fuzzing: The fuzzer operates without knowing the tested system’s internal workings. It must know the interface, but it relies on random inputs and heuristic techniques to explore the state space to identify unexpected behavior or crashes. While effective in some cases, black box fuzzing is time and resource-consuming, often resulting in incomplete coverage.
Grey Box Fuzzing: The fuzzer has some information about the internal workings of the tested system. The definition is very wide, an example is defined Invariants (see Property-based Fuzzing).
White Box: In contrast the fuzzer was provided all information about the tested system. This approach allows for more targeted and efficient fuzzing, focusing on critical code paths and specific scenarios, leading to quicker and more predictable bug detection (see Manually Guided Fuzzing).
Getting more: Advanced Fuzzing Techniques
There are more advanced techniques that allow testing in even more edge-case scenarios:
Differential Fuzzing: The fuzzer compares the outputs of the tested system with those of a reference. The reference can be a model of the system implemented in a high-level language like Python. The subject of the test could also be a single function compared to an existing library implementing the same function. Differential fuzzing is an elegant technique mainly with mathematical functions that can reveal many rounding and precision errors of Solidity and becomes very powerful when combined with stateful fuzzing. An example of a differential fuzz test is the IPOR audit, where Ackee Blockchain Security compared the implementation of continuous compound interest rate calculations in contracts to a tailor-made Python model, which yielded one critical and two high-severity issues.
Fork Testing: For testing projects with complex dependencies or integrations (Aave, Chainlink). Fork testing involves running tests on a development chain (like Anvil) that forks from a live network. This allows the system to interact with real-world data and states without needing to redeploy and mock contract storage. Using real-world data ensures more accurate and comprehensive testing than data mocking (a technique used in Formal Verification). An example of a fork fuzz test is the Lido Stonks audit, where Ackee Blockchain Security forked the Ethereum mainnet to test the protocol under realistic conditions. The use of a forked USDT contract led to an integration issue discovery of medium severity, see complete source code here.
Exploring Property-Based Fuzzing
Property-based fuzzing, or Invariant testing, falls between black box and white box fuzzing. Here, the tester (with some level of knowledge of the tested system) defines specific properties called invariants that the system must always satisfy, regardless of the inputs. The fuzzer then tries to generate inputs that violate these properties. With the tester’s introduction of invariants, the fuzzer operates with knowledge of the system’s properties, which gives the “gray-box” designation. While more controlled than black box fuzzing, property-based fuzzing still struggles with fully exploring complex state spaces, especially in stateful rich systems like complex smart contracts.
An invariant is a test that is executed after each state change (transaction).
@invariant()
def invariant_balance(self) -> None:
assert self.token.balanceOf(self.admin) == self.balances[self.admin]Introducing Manually Guided Fuzzing
Manually Guided Fuzzing combines the strengths of Stateful Fuzzing and White Box Fuzzing, adding the concept of Flows to provide a structured approach to testing. This method requires the tester to fully understand the tested system and direct the fuzzing process, ensuring that the critical parts of the code are thoroughly examined.
A Flow is a sequence of actions or transactions within the system. For instance, a flow might involve transferring tokens from one account to another. The tester defines these flows to ensure the fuzzer targets important code paths.
A flow is a single test step executed in a test sequence. Flows are defined using the @flow decorator:
@flow(precondition=lambda self: self.count > 0)
def flow_decrement(self) -> None:
self.counter.decrement(from_=random_account())
self.count -= 1How Manually Guided Fuzzing Works
Using the Wake Framework, Manually Guided Fuzzing involves several steps:
- Defining Invariants: Similar to property-based fuzzing, Manually Guided Fuzzing requires the definition of properties or invariants that should remain true after a flow is executed. For example, after a token transfer, the invariant could be that the total supply of tokens remains constant and that balances are updated correctly.
- Defining Flows: Now the tester instructs how the fuzzer should interact with the contract. If we stick to the previous example, it will initiate the token transfer.
- Combining Flows and Invariants: Manually Guided Fuzzing creates a more efficient fuzzing process by combining random flows, each flow followed by all invariant checks. Rather than randomly exploring the state space, this method allows the tester to focus on specific areas of the code, significantly reducing the time and computational resources needed.
(Dis)advantages of Manually Guided Fuzzing
- Efficiency: By directing the fuzzer with specific flows and properties, Manually Guided Fuzzing significantly reduces the explored state space and, thus, the time required to find bugs.
- Flexibility: Testers have full control over the fuzzing process, enabling them to test specific scenarios, including stateful interactions, cross-chain transactions, and complex contract dependencies, which are often difficult to cover with black box fuzzing or formal verification.
- With great power comes great responsibility: The tester must identify possible red flags in the code and decide to cover them with a fuzz test. Usually the tester defines flows for all public-state changing functions and chooses the right function’s input values. If the tester misses those red flags or does not have an attack vector idea, the fuzzing campaign will not discover the vulnerabilities.
Conclusion
Manually Guided Fuzzing represents a responsibility shift from testers using brute force or heuristic algorithms back to the hands of auditors and security researchers who have to guide fuzz tests toward attack vectors to discover vulnerabilities. Manually Guided Fuzzing offers a more efficient, precise, and scalable method for testing complex systems like heavily integrated smart contracts. Explore the mentioned techniques using the Wake framework which supports Manually Guided Fuzzing, fork testing, and differential testing.
Additional Resources
The following examples present real-world usage of different types of fuzz tests:
- Differential fuzz testing of Solidity Merkle tree against Python Merkle tree implementation – https://github.com/Ackee-Blockchain/tests-solady/blob/main/tests/test_merkle_proof_fuzz.py#L10
- Fork testing using a real USDt token instead of a mocked token – https://github.com/Ackee-Blockchain/tests-lido-stonks/blob/main/tests/test_fuzz.py#L19
- Fuzzing flows with Wake – https://ackee.xyz/wake/docs/latest/testing-framework/fuzzing/#flows
- Stateful invariant testing comparing Solidity values against Python model – https://github.com/Ackee-Blockchain/tests-lido-simple-delegation/blob/main/tests/test_fuzz.py#L605-L613
- Stateless invariants as assertions – https://github.com/Ackee-Blockchain/tests-lido-simple-delegation/blob/main/tests/test_fuzz.py#L382-L389
