How is it possible that Cardano has a serialization bug if it has specifications 🤔 Let's break down how an effort to IMPROVE the specifications actually led to this bug A lesson for any blockchain project trying to add correctness ↓

For those who didn't know, Cardano had a bug where a seemingly harmless refactoring allowed in invalid transaction to be submitted Any version of Cardano before the refactoring rejected the transaction. Any version after accepted it

Cardano uses a serialization format called "CBOR" (Concise Binary Object Representation) for its onchain data This isn't a Cardano-specific format. It's a format used by many other programs and tools since 2013

The goal of Cardano was to be build an (as much as possible) provably correct blockchain. However, all projects must start somewhere The plan: 1. Build a blockchain that works (released in 2017) 2. Build a blockchain based on specifications (released in 2020)

That means that the initial implementation of Cardano used hand-written serialization logic (in Haskell) As work was ongoing to build specifications for every part of Cardano, how would we build specifications for the serialization logic?

2019 an answer came: CDDL! A way to create clear specifications of what CBOR structures are allowed by your program

CDDL allowed the team to write a clear specification of exactly what data the Cardano blockchain accepts However, we still have a problem: how do you turn this specification into code 🤔

The CDDL project did not come with any way to convert specifications written with it into other languages It only contained a way to check if binary data matches a specification So can the gap be closed?

For the Rust implementation of Cardano, the answer was clear: generate Rust code from the CDDL spec! Cardano Serialization Lib (CSL) is the first to do this: generate serialization Rust code from CDDL CML came later with both serialization & deserialization support

But the Haskell library already did the tedious work of writing all the parsing code by hand Does it really want to spend 1~2 years writing a robust codegen tool just to achieve feature parity? Instead, the gap was closed in two steps:

1. Keep the old (handwritten) Haskell code, and write tests that match that the data it generates matches the specification (makes sense to avoid delaying shipping by 1~2 years just for a codegen tool) 2. Make the code match the spec exactly afterward

For step (2), there are two directions you could go a) Generate Haskell from the spec b) Generate the spec from Haskell To avoid subtle bugs a codegen tool could introduce, they picked option (b): generate the spec from the Haskell

Unfortunately, whenever your spec and your implementation are not the same, there is a chance something could go wrong A bug slid in where 1. It wasn't visible from the generated CDDL 2. It wasn't caught in the tests checking the Haskell code matched the generated CDDL

Hope this thread shows how decisions can make sense at every step, yet still lead to a bug It's not like other choices were clear winners. You could codegen CDDL→Haskell just to have a subtle codegen bug instead Their approach admits some downsides, but no approach is perfect

For example, if the Rust codegen approach is great, why doesn't everybody use it? It is/was used by many popular dApps and wallets, but not @Amaru_Cardano One issue: generated code isn't always a good interface. ex: took ~2ys for the CDDL→Rust code to have good Rust bindings

One of the main issues was composability: Rust → WASM doesn't tree-shake well in JS Want to compose a generated CDDL→Rust library for the ledger with a generated CDDL→Rust library for a dApp? It can lead to lots of duplicate code (larger web page size aka slower load time)

However, a stage 1 proposal to WebAssembly introduces WASM Components which could solve this composability issue It's why I've been coincidentally working on WASM Component tooling right before the attack

I still dream of a system which clear, modular specifications combined with good codegen. Something that's easy to use, secure, and performant A final shill to end the thread: that idea still somewhat lives on in Starstream. We'll have some demos on this (incl. WASM components!)