Hi all, maurelian from OP Labs here. In this post, we are sharing an internally developed framework for identifying code which we believe should be audited prior to production deployment.
Our purposes for publishing this framework are twofold:
Providing an example which other contributors to the OP Stack may take into account when deciding how to get audits.
Seeking feedback from the community. Since Optimism Governance is the ultimate decider of what gets shipped to production, it’s important to ensure that the efforts made by core contributors like ourselves align with community expectations.
The Framework
This framework provides guidance to teams at OP Labs who are writing security critical code for the OP Stack and are interested in obtaining external review, including:
A security review or audit from a security firm.
A community audit competition from a platform such as Sherlock.
A bug bounty
It will describe what code should, from our perspective, be audited, and how to prepare for an audit.
Philosophy Behind External Security Audits
It is important to underscore that security starts with the developers. An audit should not be seen as a means of “purchasing a security guarantee” from an external vendor.
What a security vendor provides is:
Feedback and insight into your ability to write secure code.
A final assurance about the specific code you have developed.
A public artifact attesting to the security of the system.
What code should be audited?
Audits are expensive and time consuming, so we recommend using them in a targeted manner.
Generally, audits are better suited for some kinds of codebases, specifically those where Safety is of Existential importance.
The following rubric divides projects along Liveness vs. Safety and Reputational vs. Existential axes.
audit-rubric 2282 × 1940 303 KB
The rubric above is not exhaustive, but should give a sense of the framework OP Labs uses to determine what requires an audit.
Most OP Stack code fits into one of the following buckets:
Secure via testing and real world usage:
This applies largely to infrastructural code, referring to components such as op-node, op-geth, op-batcher as well as alternative implementations of such components.
In general, audits are less valuable or necessary for infrastructural code because:
Experience has shown us that auditors provide less value with infrastructural code, especially relative to what is learned by getting code running on a testnet or putting code in users’ hands.
When issues do occur in infrastructural code, they can be addressed through ‘social consensus’, ie. quickly disseminating new software with a bug fix.
Secure with extreme caution (including auditing):
This applies largely to smart contract code, in particular in cases where assets are being secured.
Examples include modifications to bridge code, changes to deposit derivation code, multi-sig wallet modifications, etc.
In general, audits are necessary for smart contract code because damage resulting for vulnerabilities is more likely to be irrecoverable.
There are likely to be situations where smart contract code does not require an audit, and where infrastructure code should receive an audit. Every case requires a fact-specific analysis about whether an audit should be required.
And while there is no “magic formula” that works in all cases, analysis according to the Existential vs. Reputational x Liveness vs. Safety framework is more important than simply differentiating between infrastructure and smart contract code.
Preparation for external security review
Once again, secure code starts with the development teams that write it. We recommend completing the following steps for a given codebase prior to seeking external review.
Enumerate the invariants and security properties of the system
This step can be completed however the development team wishes, though teams should note that it is an important input to the next two items.
Documentation for security researchers
Security specific docs which describe the high-level security properties of the system. A good example of this documentation is what OP Labs provided to the Sherlock auditors 11 for the Bedrock audit.
Documentation for security researchers should outline:
The behavior that the system is expected to maintain.
The negative situations the system should prevent.
Known issues which are considered low risk and are out of scope.
Test coverage assessment
Developers should evaluate coverage of the system from two angles:
Code coverage
For each testing method, a test coverage report should be generated. This report should be reviewed to identify any gaps in coverage.
We should aim for greater than 100 % coverage of critical code paths. We do not need to be sticklers about testing trivial code paths like basic getters and such.
Property coverage
For each security property, a qualitative assessment of the test coverage should be done. It should answer the following questions:
Has this property been thoroughly tested?
Which testing methods have been used to test this property? Are there other testing methods we could apply with a reasonable amount of effort?
Are there any edge cases relevant to this property which are not fully tested?
Page 13 and onwards of this Trail of Bits report 9 may serve as a useful template for this assessment.
Conclusion
We believe that this framework provides a strong foundation for both development teams and governance in evaluating the necessity of an external review OP Stack code.
