{"id":915,"date":"2024-09-12T11:08:38","date_gmt":"2024-09-12T09:08:38","guid":{"rendered":"https:\/\/ackee.xyz\/blog\/?p=915"},"modified":"2024-09-13T16:29:39","modified_gmt":"2024-09-13T14:29:39","slug":"introducing-manually-guided-fuzzing-a-new-approach-in-smart-contract-testing","status":"publish","type":"post","link":"https:\/\/ackee.xyz\/blog\/introducing-manually-guided-fuzzing-a-new-approach-in-smart-contract-testing\/","title":{"rendered":"Introducing Manually Guided Fuzzing: A New Approach in Smart Contract Testing"},"content":{"rendered":"

Fuzzing is a well-known <\/span>software testing technique, especially useful for testing<\/span> smart contracts. Traditionally, as fuzzing, testers understood mainly <\/span>Black Box Fuzzing<\/b> or, more advanced, <\/span>Property-Based Fuzzing<\/b>. However, a new and innovative approach, <\/span>Manually Guided Fuzzing<\/b>, offers enhanced efficiency and effectiveness, filling the gaps left by traditional fuzzing techniques. This article will explore how Guided Fuzzing differs from Black Box Fuzzing, Property-Based Fuzzing, Differential Fuzzing and other techniques.<\/span><\/p>\n

Understanding the Basics: Stateful vs. Stateless and Black Box vs. Grey Box vs. White Box Fuzzing<\/span><\/h2>\n

Fuzzing techniques can be classified based on their statefulness and the level of guidance provided:<\/span><\/p>\n

<\/h3>\n

Stateful vs. Stateless Fuzzing<\/b>:<\/span><\/h3>\n

Stateless Fuzzing<\/b>: Each test case is independent of the others in stateless fuzzing. The fuzzer does not maintain any knowledge of previous states or inputs, which limits its ability to discover bugs related to the sequence of operations.<\/span><\/p>\n

Stateful Fuzzing<\/b>: This method considers the system’s state, meaning the outcome of one test case can influence the subsequent ones. Stateful fuzzing is more effective for testing complex systems like smart contracts, where state transitions are critical. See the difference on <\/span>this code snippet<\/span><\/a> written in <\/span>Wake Framework<\/span><\/a>.<\/span><\/p>\n

<\/h3>\n

Black Box vs. Grey Box vs. White Box<\/b>:<\/span><\/h3>\n

Black Box Fuzzing<\/b>: The fuzzer operates without knowing the tested system’s internal workings. It must know the interface, but it relies on random inputs and heuristic techniques to explore the state space to identify unexpected behavior or crashes. While effective in some cases, black box fuzzing is time and resource-consuming, often resulting in incomplete coverage.<\/span><\/p>\n

Grey Box Fuzzing<\/b>: The fuzzer has some information about the internal workings of the tested system. The definition is very wide, an example is defined <\/span>Invariants <\/b>(see Property-based Fuzzing).<\/span><\/p>\n

White Box<\/b>: In contrast the fuzzer was provided all information about the tested system. This approach allows for more targeted and efficient fuzzing, focusing on critical code paths and specific scenarios, leading to quicker and more predictable bug detection (see Manually Guided Fuzzing).<\/span><\/p>\n

<\/h3>\n

Getting more: Advanced Fuzzing Techniques<\/b><\/h3>\n

There are more advanced techniques that allow testing in even more edge-case scenarios:<\/span><\/p>\n

Differential Fuzzing<\/b>: The fuzzer compares the outputs of the tested system with those of a reference. The reference can be a model of the system implemented in a high-level language like Python. The subject of the test could also be a single function compared to an existing library implementing the same function. Differential fuzzing is an elegant technique mainly with mathematical functions that can reveal many rounding and precision errors of Solidity and becomes very powerful when combined with stateful fuzzing. An example of a differential fuzz test is the IPOR audit, where Ackee Blockchain Security compared the implementation of continuous compound interest rate calculations in contracts to a tailor-made Python model, which yielded one critical and two high-severity issues.<\/span><\/p>\n

Fork Testing<\/b>: For testing projects with complex dependencies or integrations (Aave, Chainlink). Fork testing involves running tests on a development chain (like Anvil) that forks from a live network. This allows the system to interact with real-world data and states without needing to redeploy and mock contract storage. Using real-world data ensures more accurate and comprehensive testing than data mocking (a technique used in Formal Verification). An example of a fork fuzz test is the Lido Stonks audit, where Ackee Blockchain Security forked the Ethereum mainnet to test the protocol under realistic conditions. The use of a forked USDT contract led to an integration issue discovery of medium severity, see complete source code <\/span>here<\/span><\/a>.<\/span><\/p>\n

<\/h3>\n

Exploring Property-Based Fuzzing<\/b><\/h3>\n

Property-based fuzzing<\/b>, or <\/span>Invariant testing<\/b>, falls between black box and white box fuzzing. Here, the tester (with some level of knowledge of the tested system) defines specific properties called <\/span>invariants<\/b> that the system must always satisfy, regardless of the inputs. The fuzzer then tries to generate inputs that violate these properties. With the tester’s introduction of invariants, the fuzzer operates with knowledge of the system’s properties, which gives the “gray-box” designation. While more controlled than black box fuzzing, property-based fuzzing still struggles with fully exploring complex state spaces, especially in stateful rich systems like complex smart contracts.<\/span><\/p>\n

An invariant is a test that is executed after each state change (transaction).<\/span><\/p>\n

@invariant()\ndef invariant_balance(self) -> None:\n\u00a0\u00a0\u00a0\u00a0assert self.token.balanceOf(self.admin) == self.balances[self.admin]<\/code><\/pre>\n
Introducing Manually Guided Fuzzing<\/span><\/h2>\n
Manually Guided Fuzzing<\/b> combines the strengths of Stateful Fuzzing and White Box Fuzzing, adding the concept of <\/span>Flows<\/b> to provide a structured approach to testing. This method requires the tester to fully understand the tested system and direct the fuzzing process, ensuring that the critical parts of the code are thoroughly examined.<\/span><\/p>\n
A <\/span>Flow<\/b> is a sequence of actions or transactions within the system. For instance, a flow might involve transferring tokens from one account to another. The tester defines these flows to ensure the fuzzer targets important code paths.<\/span><\/p>\n
A flow is a single test step executed in a test sequence. Flows are defined using the @flow decorator:<\/span><\/i><\/p>\n
@flow(precondition=lambda self: self.count > 0)\ndef flow_decrement(self) -> None:\n    self.counter.decrement(from_=random_account())\n    self.count -= 1<\/code><\/pre>\n
How Manually Guided Fuzzing Works<\/span><\/h3>\n
Using the Wake Framework, Manually Guided Fuzzing involves several steps:<\/span><\/p>\n
    \n
  1. Defining Invariants<\/b>: Similar to property-based fuzzing, Manually Guided Fuzzing requires the definition of properties or invariants that should remain true after a flow is executed. For example, after a token transfer, the invariant could be that the total supply of tokens remains constant and that balances are updated correctly.<\/span><\/li>\n
  2. Defining Flows<\/b>: Now the tester instructs how the fuzzer should interact with the contract. If we stick to the previous example, it will initiate the token transfer.<\/span><\/li>\n
  3. Combining Flows and Invariants<\/b>: Manually Guided Fuzzing creates a more efficient fuzzing process by combining random flows, each flow followed by all invariant checks. Rather than randomly exploring the state space, this method allows the tester to focus on specific areas of the code, significantly reducing the time and computational resources needed.<\/span><\/li>\n<\/ol>\n
    <\/h3>\n
    (Dis)advantages of Manually Guided Fuzzing<\/span><\/h3>\n