Orignal title: How I Learned to Stop Worrying & Love Execution Sharding
Video link: https://www.youtube.com/watch?v=A0OXif6r-Qk
Lecturer: Scott Sunarto (@smsunarto) on Research Day
Article edited by Justin Zhao (@hiCaptainZ)
Hello everyone, my name is Scott (@smsunarto), the founder of Augus Labs (@ArgusLabs_). Today, I'm going to discuss a topic we haven't touched on for quite some time. As roll-ups have become the zeitgeist, we don't often talk about execution sharding as much as we do about data sharding. So, let's revisit this somewhat neglected topic of execution sharding.
This will be a relaxed talk. I know everyone has been listening to complex concepts all day, so I'll aim to keep this as practical as possible. I've prepared a proper slide design for this presentation.
For those who don't know me, a fun fact is that I'm known as an anime girl on Twitter. I also missed my college graduation to be here, much to the chagrin of my parents. Currently, I'm the founder at Argus Labs. We identify as a gaming company, not an infrastructure or crypto company. One of my biggest gripes is that everyone in crypto games wants to build tools, but no one wants to create content or applications. We need more applications that users can actually use.
Previously, I was the co-creator of Dark Forest (@darkforest_eth) with my brilliant friends Brian Gu (@gubsheep) and Alan Luo (@alanluo_0). Brian is now running 0xPARC (@0xPARC) and is many times smarter than me.
Today's talk will focus on execution sharding, but in a context that is unfamiliar to most of the discourse we have about execution sharding. We usually discuss execution sharding in the context of a layer one, like Ethereum sharding or Near sharding. But today, I want to shift the context a bit. Let's think about how sharding would look in a roll-up environment.
The underlying question here is why a game company is architecting their own roll-up and what we can learn from World of Warcraft in designing roll-ups. Also, we'll explore how the design space for roll-ups can expand way beyond the current reality.
To answer these questions, let's go back in time to 2020 when the idea for Dark Forest was first conceived. We asked ourselves, what if we created a game where every single game action is an on-chain transaction? It was an absurd premise back then and still is to many people today. But it was an interesting hypothesis, so we built it, and Dark Forest was born.
Dark Forest is a fully on-chain space exploration MMORTS game on Ethereum, powered by ZK-Snarks. Back in 2020, ZK was not as popular as it is today because documentation was pretty much non-existent. The only available documentation for Circom was Jordi Baylina's (@jbaylina) Google Docs. Despite the challenges, we learned a lot from the process, and Dark Forest is the manifestation of that.
Dark Forest was an experiment that grew bigger than we thought it would. We had over 10,000 players, trillions of gas spent, and a lot of in-game chaos with people backstabbing each other on-chain. The most fascinating thing about Dark Forest and on-chain games is the nature of platformization. By having an on-chain game, you open up the design space for emergent behavior and allow people to build smart contracts that interact with the game, and alternate client and game modes,for example Dark Forest Arena and GPU miners.
However, with great power comes great responsibility. When we launched Dark Forest on xDai, which is now known as Gnosis Chain, we ended up filling the entire block space of the chain. This made the chain pretty much unusable for anything else, including DeFi, NFTs, or any other xDAI stuff.
So, what now? Are we at a dead end? Will on-chain games never be a reality? Or do we go back to making games where the only thing on-chain is JPEGs and convince people that money grows on trees? The answer is, we make software do things. Many of us think about blockchains and roll-ups in a very rigid way, as if there's not much room for improvement. But I disagree. We can experiment and find out new possibilities.
We asked ourselves the question: if we were to design a blockchain from the ground up for gaming and gaming only, how would it look? We need high throughput, so things need to scale both reads and writes. Most blockchains are designed to do a lot of writes. Transaction per second is the metric that people tout, but the reality is, reads are equally important. How would you know where to play your locations are if you can't read from a blockchain node? This is actually the first bottleneck that we found within blockchain constructions.
Dark Forest encountered an issue with full notes being heavily utilized and I/O exploding because we needed to read data from the on-chain state. This resulted in thousands of dollars in server costs, which the xDAI team graciously covered for us. However, this is not ideal for the long term. We need high throughput not only for write transactions per second but also for reads, like getting data from the blockchain itself.
We also need a horizontally scalable blockchain to avoid the Noisy Neighbor problem. We don't want a popular game to suddenly start falling on a blockchain and stop everything from working. We also need flexibility and customizability so we can modify the state machine to be designed for games. This includes having a game loop, making it self-executing, and so on.
Last but not least, and this may be somewhat vague for those unfamiliar with online game architecture, we need a high tick rate. Ticks are the atomic unit of time in a game world. In a blockchain context, we have blocks as the atomic unit of time. In games, we have ticks. This is pretty much analogous when you're building a fully on-chain game where the speed of your blockchain ticking or the speed of block production is equal to the tick of the game itself.
If you have a higher tick or a higher block per second, the game feels more responsive. Conversely, if you have a lower tick, the game feels more sluggish. One key thing to keep in mind is that if block production is delayed, you will visually feel a lag in the game. This is a terrible experience. If you've ever dealt with an angry gamer screaming at a computer because they lost a game, it's an absolutely terrible situation to deal with.
Currently, our rollups have one block per second, which is equal to one tick rate. If we want to have cooler games, we need higher tick rates. For example, Minecraft, a simple pixel art game, has 26 ticks per second. We have a long way to go to build games that are as responsive as Minecraft.
One possible solution is deploying our own rollup. While it seems like it solves the problem on the surface level, it doesn't actually solve the underlying root cause of the problem. For instance, you would have higher throughput for writes, but not quite at the level that a game needs. Sure, if your games have a hundred players, it will be enough. But if you want to build games that require much higher throughput, there are very strict limitations because of how I/O is done in current constructions.
In terms of reads, you don't really get the performance boost. You still need to rely on indexers. You don't really have horizontal scalability. If you try to spin up a new rollup to horizontally scale your game, you're fragmenting your existing smart contract ecosystem. Marketplaces that players deploy wouldn't work with the other chains where you've spun up to horizontally scale your game. This opens up a lot of problems.
Lastly, high tick rate and blocks per second are just not a thing. We can try and push it as hard as we can. We might get two blocks per second, maybe three, but that's really as far as these blockchains are going to go because there are a bunch of overheads like re-marshalling and stuff that are very taxing on compute cycles.
To address this, we look back to the early 2000s and late 1990s when online games like MMOs were just coming up. They had this concept of sharding. This is not a novel concept; it has existed in the past. The term "sharding" that we use in database architectures actually comes from references in Ultima Online. They were one of the first to use the term "sharding" to explain their different servers.
So, how does sharding work in games? It's not a one-size-fits-all solution. It's a tool in the toolbox, and how you fit it in your game differs from context to context. For instance, the first sharding construction is what I like to call location-based sharding. A good mental model to think about it is to imagine a Cartesian diagram split into four quadrants, each quadrant having its own game shards. Every time you want to cross a shard, you make a communication to the other shard saying, "Hey, I want to move there," and then you get teleported to your shard, leaving your previous player body behind. By doing this, you're distributing the workload of the server into multiple physical instances instead of forcing a single server to do all the computations for the entire game world.
The second construction is more popular these days. It's called multiverse sharding, where you have multiple instances of the game mirrored with each other. You can choose whichever shard you want to go to, and it's load-balanced by default, such that each server is not getting too congested.
Now, the key question here is how do you bring this concept to a rollup? This is why we created the World Engine. The World Engine is our flagship infrastructure, which is basically an opinionated shard sequencer designed for starting. In contrast to a lot of the shard sequencer designs that we've seen in the past few talks, our design is different and more suited to our needs. What we are optimizing for is: A, throughput, and B, we want to make sure that there is no locks that is blocking the runtime to make sure that the tick rate and block time is as efficient as possible and so again it's default synchronous and the way that we design a sequencer is such that it's partial order instead of like force total ordering where like you know each transaction needs to happen after the other.
The key component here is that we have two main things. We have the EVM-based shard, where it's like a pure EVM chain where players can deploy smart contracts to compose with the game, create marketplaces with taxes, and so on. It's going to be like a normal chain, right? Like one block per second or one thing per second, just enough for you to do all your typical device and marketplace stuff.
The secret ingredient here is that we use a game shard, which is essentially a mini blockchain designed to serve as a high-performance game server. We have a bring-your-own-implementation interface such that you can customize this shard to your liking. You can build your own shards to inject into the base shard. You just need to implement a standard set of interfaces, like the same way that, if you're familiar with Cosmos, Cosmos has an ABC interface. You can basically conform this to a similar spec to bring your own shards to the world engine stack.
The key thing here is that we have a high tick rate that we currently can't achieve with current sharding constructions. This is where I want to introduce Cardinal. Cardinal is the world engine's first game shard implementation. It uses the entity-component-system, which has a data-oriented architecture. This allows us to parallelize the game and have higher throughput for the game computations. It has a configurable tick rate of up to 20 ticks per second. For the blockchain people here, that's 20 blocks per second.
We can also geolocalize this to reduce latency. Right now, you have sequencers that may be based in the US, and then people in Asia have to wait, like, 300 milliseconds latency before that transaction hits the sequencer. This is a huge problem in games because 300 milliseconds is a long time. If you try to play an FPS game with 200 milliseconds latency, it's essentially forever, and you're already dead.
Another key thing here that's also important for us is that it's self-indexing. We don't need to have an external indexer anymore. We don't need these frameworks to cache the in-game state. This also allows us to build more real-time games that don't have latency issues because the indexer is still trying to catch up with the sequencer blocks.
We also have a plugin system that allows people to parallelize ZK verification and so on. The best part of all, at least for me, is that you can write your code in Go. No more wrangling with Solidity trying to make your game work. If you've tried to build a blockchain game in Solidity before, it's an absolute nightmare.
But yeah, the key thing here with our shard construction is that you can build anything as shards. They're just like basically an infinite design space of like what a shard can be.
Let's say you dislike writing your game code in GO, you can opt out, but again, we're working on a solidity game shard that allows you to write your game implementation in solidity while preserving many of the benefits of Cardinal. You could also create an NFT minting shard with a unique mempool and an ordering construction that will solve issues like the basic minting Noisy Neighbor problem. You could even have a game identity shard where you can use NFTs to represent your game identity, allowing you to trade your game identity easily using NFTs instead of having to give your private key, which is not advisable.
This is a high-level construction, and I won't dive too deep due to time constraints. The key thing here is that we allow EVM smart contracts to compose with the game shards by using custom pick and pass. We create a wrapper around Geth to allow communication between them. This opens up a lot of design space bi-directionally. We're synchronous by default and have seamless composable interoperability across shards without locks.
Our shared sequencer is different in that it doesn't use a shared sequence construction that prioritizes total ordering Atomic bundles, which requires a locking mechanism and leads to issues like blocking the main thread, leading to unreliable tick rate and block time, which results in game lags. It also imposes restrictions on block time for each shard and requires various crypto economics and constructions to prevent denial of service. There's also the big elephant in the room that I haven't seen mentioned with a lot of VCR sequencer constructions: how do you resolve Deadlocks if you have different shards that are dependent on each other and cause a deadlock? With asynchronous design, none of this is a problem because everyone is just doing whatever they want to do and fire and forget.
The reality is atomic bundles across shards and Roll-Ups are often not necessary. For our use cases, we don't need anything that requires atomic bundles at all, and we don't think it's something we should engineer our Roll-Ups around for the sake of use case purity. This also leads to a lot of other interesting properties. For instance, each game shard can have a separate DA layer for the base chain. For example, you can use the base shard to push data to Ethereum, and the game shards can push data to Celestia like a data availability committee. You can also reduce the hardware requirements for running full nodes because you can run base shard Geth full nodes separately without having to run the game shard node, which makes it easier for you to integrate with things like Alchemy.
To wrap things up, I want to be intellectually honest here. A lot of people want to claim that their construction solves all life's problems, which is just not the case for us. We think our construction is useful for us, but maybe it's not for your use case. It's not realistic to assume that our construction will work for everyone's use cases. It ticks the boxes for us, gives us high throughput, horizontal scalability, flexibility, and high tick rates, but it doesn't cure cancer. If you need a DeFi protocol that requires synchronous composability, then this construction might not be the right one for you.
In conclusion, I really believe in the concept of a human-centric blockchain architecture. By designing around a specific user persona and use case in mind, you can better navigate the trade-off space instead of trying to solve everyone's problem. The era of the Renaissance is here, and everyone can design their own Roll-Ups to meet their specific needs instead of relying on generalized solutions. I believe we should embrace the Cambrian explosion. Don't build roll-ups like a one-size-fits-all layer one because it's simply not designed to solve the same problem. I'm personally excited to see more people explore more Roll-Up design spaces that are use-case or sector-specific. For instance, what would a Roll-Up designed solely for asset exchange look like? Would it be intent-based? What would a Roll-Up designed for an on-chain CLOBs look like? With that said, I'll pass the mic back to MJ. Thank you for having me.