Custom Netcode for a Battle Royale

In 2026, I was working at Funktronic Labs when they released the game Scramble Knights Online (as an Early Access title). We started working on this game as early as February of 2022.

Scramble Knights Royale eventually became Scramble Knights Online.

In those early days, we were prototyping various game modes. After a lot of iteration, the team decided on making a Battle Royale. The goals of this game were ambitious:

  • 64 player matches, potentially everyone vs everyone.
  • Physics based gameplay, using hitbox style melee fighting.
  • Proximity based voice chat, for maximum chaos.
  • Dedicated game servers, transiently scaling in and out as player counts ebbed and flowed.
  • Allow self-hosting custom matches (maybe modded?)

I spent some time researching various networking solutions for Unity, but felt dissatisfied by many of them. Being as crazy as we were back then, I opted for writing a totally custom networking solution for Unity from scratch.

I'll spoil a bit: The game did not launch as a Battle Royale, but rather it was pivoted into an MMO-lite (sort of a Monster Hunter-like) right before hitting Early Access. Thankfully, all of the networking stuff carried over and even further evolved to meet the new goals.


Where to start?

For big tasks like this, I like to split up stuff into chunks and work out each chunk at a time. Knowing that we wanted dedicated servers (including player-hosted), it was a classic server/client setup.

  • Server would be always-on, listening for players to connect.
  • Clients would somehow know the IP:Port to connect to, and request to do so.
  • Server would respond to these clients with some simple handshake, exchanging information about the map and game settings. This would include all of the information clients would need about each other as well.
  • Clients would reply with information about their player, such as number of splitscreen participants, customization loadouts, that sort of thing.
  • Server would spawn the new clients players, and relay that information to everyone else already connected.

There were certain properties about the networking that became obvious, given the overall mechanical intentions of the game too:

  • Things were bound to be fast paced, physics-oriented. This meant we very much needed to handle unreliable (UDP) streams of data.
  • Players had big blocks of data they would need to send, so of course we also needed to handle large in-order streams of reliable (TCP) data.
  • We wanted players to be able to communicate with voice via proximity chat. This could be handled unreliable or reliably, but either way needed a dedicated stream that is okay to fail on it's own, without blocking actual gameplay (TCP or UDP).

These various streams of data meant that clients and the server each need to allocate various buffers of memory. How much exactly is somewhat arbitrary and so I'll gloss over details like that, but let me break down how things were laid out.

From the perspective of a game server, every client each had their own set of streams (Reliable in, Reliable out, Unreliable in, Unreliable out, Voice in, Voice Out) plus some sort of scratch buffer for staging into and out of raw sockets. 

Seeing that this data was essentially the same for the client and server, I wrapped this up in structure that looked similar to this:

Each 'player' (note: the server is considered a 'player' from the perspective of a client). NetworkPlayers contain all of the buffers allocated for any other player, both their inputs and outputs. 
NetworkBuffers contain their raw byte array, but also read/write indices, so they can be written to and read from in various stages of the networking pipeline. 

The reason for splitting the buffers into all these inputs and outputs, with read/write cursors, is so we can think about things asynchronously as much as possible. Consider the following example CPU threads:

In theory, it's most optimal to read in network data before the GameUpdate, and then write out any outgoing data after GameUpdate, such that we never stall the main thread with networking work.

In practice of course, the server will have various network threads (a simple static thread pool is fine for 64 players, where the main thread sets a flag 'gooo!!!' and the worker threads grab the work that needs to be done and then idle until more work is available. That looks something like this:


With this architecture in mind, I believed it would be easiest to have network messages simply flow in and out of the "networking layer" as raw bytes, while the "game layer" only sent in and got back out user-friendly structured messages.

Keeping these worlds entirely distinct in this way would allow me to focus on optimizing each piece on it's own, and keep most of the complicated networking bits hidden away from the other developers. Hopefully, this would lead to less bugs as the game was developed further. So, I started working out the game side of it first: a NetworkMessage.

Network messages would need to somehow be serialized, then those serialized bytes would be sent to other players and back. When receiving those bytes, they'd get deserialized back into network messages.

The hope was that the game would not necessarily need to care about where most messages came from and would allow each message to do whatever it needed on the event one came in.

Network Messages

I wanted network messages to be extremely easy to write, to minimize the amount of bugs generated by other developers and myself. We're only human, and we're very prone to making mistakes.

So, I started with a simple Interface.

This interface was kept as simple as possible. Tag a struct as a network message and give it some callbacks for the client and server separately. This allowed either to handle each message on a case by case basis. 

Using this interface, the very first NetworkMessage was a simple ping.

You might notice, there's a [GameNetworkMessage] attribute in there, too! As I was writing this first network message, it dawned on me that I absolutely did not want to hand write serialization/deserialization for every single network message by hand. The whole point of this system was to keep things easy to write for other developers.

So, my solution was to tag network messages with an attribute, then generate code at build time for automatically serializing and deserializing the public fields in any particular network message.

To start the prototype off simple, I simply hooked this step up into a button in the tools menu of Unity. You'd hit "Game -> Refresh Networking" and all network messages would automatically generate InternalNetworkMessage structure variants with serialization/deserialization written out for you.

The auto-generated code would result in something like this:

Note: I did not expect other developers to be conciously aware of this internal-network-message layer, but it was visible and ready for easy debugging if necessary.

The code that generates code is fairly straightforward, it simply looks over structs that have the [NetworkMethod] attribute and fills out the text in a giant InternalNetworkMessages.cs file.

Then, using a network messages becomes as simple as:

At the time, this felt like a good solution to me. Write a simple network message, construct it, send it, that's it. Unfortunately I later came to realize that this was simply too many steps for most developers (even myself) as often things would go wrong somewhere in this process. (I'll get back to this, later.)


The bridge between NetworkMessages and bytes

At this moment in development, I was pretty happy with the game-side of things, the way network messages were kind of abstracted away from the actual networking that needed to be done. But now, I needed to actually start generating bytes and shuffling them around the internet.

Just a moment ago, I teased this method: NetManager.SendMessage(). Inside of here is where the transition from a structure into raw bytes happens. In a nutshell, there is a conversion from NetworkMessage to InternalNetworkMessage, and the Internal* variants contain the Read/Write buffer methods. But what would be to serialize things? There are plenty of libraries that do this, especially in c#. However, I do like to try and minimize dependencies where reasonable. So, I decided on writing serialization methods, and wanted to ensure that it would be easy to slot in more in the future.

To write a serialization/deserialization pair as simply as possible, I used attributes. I made NetworkSerializeWriterAttribute and NetworkSerializeReaderAttribute, which could be tacked onto any static method to designate it as a reader or writer for any struct or class.

This worked for basically any type. When "refreshing" the network code, it would automatically generate calls to these methods based on the types inside of a NetworkMessage.

I wrote out the read/write for all the base types in c# ( int, string, etc), as well as most of the Unity types (Vector3, Quaternion, etc).


A Problem

I loved this solution, on some sort of primal personal level. I like writing code, and I like knowing exactly where bytes are coming from and where they are going. However, not everyone is wired like this. When other engineers on the projected wanted to contribute to anything networking related, they often stumbled on this step in particular. Writing serialization, writing network messages, refreshing, remember which API to use. It was too much!

So I went back to the drawing board on this. I asked the other engineers what they actually wanted to do, most of the time. What they really wanted was pretty simple: "I want to call a method for all players." Almost always, that's all they wanted. Admittedly, thinking about it in hindsight, this should have been obvious. With this in mind, I thought about how I could change the workflow in order to allow them to do this more easily.

The solution I ended up with, I am very happy about even today: tag a method with a [NetworkMethod] attribute, and then everything is somehow magically handled for you. You can call that method, and it "just works" for all players in a match.

This was possible because of two things:

  • There was already a 'refresh netcode' button, which was free to generate any necessary code.
  • The game is made in Unity, which uses C#, which has modifiable object code after compilation.

I set it up such that code that looks like this:

..would "just work". The steps to get there:

  1. Write out a method with an attached [NetworkMethod] attribute.
  2. Hit the refresh netcode button in unity (whenever finished).
  3. a NetworkMessage would automatially be built for you, somewhere hidden away.
  4. An InternalNetworkMessage would automatically be generated from that, with serialization of parameters handled for you automatically too, using the methods above.
  5. The compiled assembly ( dll ) would be 'weaved' automatically, replacing all instances of that method being called with the previously auto-generated NetworkMessage, feeding it through the appropriate API to send the message.

This made the code a bit of a lie. However, this sped up development of anything netcode-related dramatically. The best part is that there really was not any overhead to the runtime for this, it really was doing the same work as manually writing all the code anyways. Much less bug prone, as a cherry on top.


Bringing it all together.

At this point, we have a bunch of parts:

  • A work scheduler, for the server to communicate with various clients at once without stalling the main thread.
  • A system for writing NetworkMessages manually
  • A system for writing NetworkMethods manually, which auto generates NetworkMessages.
  • A system for adding serialization/deserialization for any custom types.

These bits were all the foundation of the networking plugin, and both individually and together everything seemed to work great. That is, until we started getting a lot of players in one area.

Even if these systems are great and fast at a fundamental level, the way code is written still matters a great deal. Most aspects of the initial prototype were very 'naive' in the sense that they did things at the most basic level. For example, players position updates would send their updates to every other player. And at a scale of 64 players, that's the server sending 64 * 64 = 4096 rigidbody updates every server tick (1/30), so 122,880/s * 76 bytes = ~9,000,000 bytes/s, or 9 MB/s. Not impossible, but a pretty insane number for JUST player updates.

Getting this number down to a more reasonable level could be achieved in a few ways. The first, most obvious one, is to simply crunch down the precision of the types being used in updates like this.

Rotations don't need to be full quaternions for basically anything, in this game in particular, so just send the axis Y rotation. Velocities really do not need to be full precision in basically any scenario, so crunch them down. Unfortunately, positions really do need to be higher precision or you start losing visual quality pretty drastically, so those were left alone. These little changes got us from 72 bytes to 50 bytes, not bad! That takes us from about 9MB/s to 5MB/s, which is still way too high. Something needed to change at a fundamental-systems level.

Taking everything apart.

Running through the project's NetworkMessages and crunching compressions was definitely worth doing, but it was not going to be enough as a long term solution to keeping bandwidth low. I needed to find a way to be able to determine if any particular message was actually worth sending to any other particular player. At 64 players it may not seem unreasonable to simply do a basic distance check "is this player too far from me? skip sending the message", but ~4,000 distance checks for a single message, and the fact there's hundreds of messages being send per frame, means we may potentially be performing millions of (redundant) distance checks over and over.

One early idea I had was to simply cache them, but even performing 4,000 distance checks per frame seemed a bit wasteful to me, and in practive it still turned out to be pretty slow (because it turned out it wasn't just distance checks for rigidbody updates, but you really need arbitrary checks from any position to any player, as you do not know what will generate network messages ahead of time).

Because of some past experience with voxel tech, I tend to think about everything in terms of 'chunks' or 'buckets,' so when I thought about this problem an interesting solution came to me: Split the world into distinct chunks, then do a distance check for every player to every chunk, once per server tick. Then, if a network message is generated, only send it to players within range of the chunk it originated from!

If player A is very far from player B, they'll be in far away 'chunks' in the world. In this case, any NetworkMessageRigidbodyUpdate that would be generated by A, would never reach B.

My first implementation of this system was successful, although it was a bit slow. Luckily, this kind of problem was perfect for the Unity Job System, and so I rewrote it as a multi-step job.

  • Gather a list of game client's player positions.
  • Put them into 'buckets' or 'chunks', by taking their position (float3) and dividing it by some arbitrary value (chunk_size). Clamp it into an int3 and use that a key into a Dictionary<int3, Chunk>
  • For every player, maintain a list of visible_chunks (a List<int3>). The list would simply contain their current chunk, and all touching neighbor chunks (the 3x3 grid, essentially).
  • When any NetworkMessage tries to SendAll(), when iterating over players it will check if the NetworkMessage's origin position's chunk is in the list of visible_chunks for each player. If not, the player is skipped.

Because of how the game was structured, this was a huge huge win. Typical network usage dropped from the 10s of MB down to KB per second, when idle.

Going even further beyond

At this point, I was very happy with the results. But, KB/s was still really not good enough. I kept thinking to myself "surely that's too many bytes." "What is actually being sent?" was another thought that often popped into my head. So, at some point I finally started logging the NetworkMessages and seeing what was really eating up all that bandwidth.

I expected the number 1 thing to be rigidbody updates, since they are so frequent and there are a lot of synced rigidbodies. But, to my surprise, it was actually 'visibility updates'. These were a result of the chunked system I described above: when a player crossed a chunk boundary, the server kept track of what objects were in which chunks, and so it would send a batched "destroy these objects" and "create these objects" to that player. This kept the object count low for players, ensuring they were not wasting time with offscreen objects. But, players run around the world a lot. And these messages added up fast, mostly because they contained string GUIDs of prefab references into the asset bundle system. "spawn tree" "destroy crate" etc.

Seeing this, I felt like it could be resolved if the strings themselves could easily be compressed. After all, they're just GUIDs right? They shouldn't eat up that much space. I think this would have been a valid pursuit, but I actually ended up doing something a bit more generalized: I compressed the entire network stream.

My first attempt at compressing the stream was very basic: run length encode (RLE) the stream, byte by byte. This did work, but I did not see much of a gain out of it. Bytes were just too random! So I printed out a chunk of the network stream as bytes and stared at it for awhile.  

After looking at the raw byte stream for awhile, I was thinking about what is actually in the bytes most of the time. The raw stream always contained this, just over and over:

The size is 2 bytes, ID is 2 bytes, contents are variable.

And the contents themselves contained data that often looked like this:

strings were an int (4 bytes) size, then int (4 bytes) char, char, char.

If you squint, you may notice that almost everything here is either 4 bytes or 2 bytes. Nothing of consequence is JUST one byte. So I had an idea, why not run RLE but over 4 bytes at a time?

This actually worked very well! I started to see ~50% less network bandwidth being used across the board. Often, idling was as small as ~500 bytes per second.

This was pretty good, I thought. But, if I was able to do this by just thinking about it, I'm pretty sure someone smarter had a much more elegant solution. I did a little research about how Unity handles compression in general and came across LZ4.

I won't get into the details of LZ4 compression, because, franky, I have absolutely no clue. It's basically a magic black box written by some genius, from my perspective. I found a single file c implementation of it, then ported it into c# and made that code compatible with Unity's Burst compiler so it could be run as a basic static method within my network jobs.

My first test of this was an even biggest success than my silly 4-byte RLE test, I saw 80%-90% reductions in bandwidth across the board. 👏Great, I was very satisfied with this. LZ4 was even pretty good at compressing seemingly random data, such as rigidbody updates and voice packets.

Running around the overworld with many other players, uploading ~300bytes/s downloading ~4.6kb/s

With these two methods combined (chunking the world for visibility culling and network stream compression), bandwidth was in a shippable spot.

After this, I spent some time on abstracting out the "Transport Layer" of the plugin, so the game could support using Epic Online Services as a transport backend for p2p connections and also support raw c# sockets for typical dedicated servers. This was pretty straightforward, more or less what you'd expect, so I won't ramble about it here.


At this point, I felt the "networking plugin" aspect was more or less complete. Everything that came after was moreso about how to effectively engineer the game itself, without having to think too hard about netcode awhile doing so.

Show Comments