Hitting 90z on the Quest 2
In 2023, I worked at Funktronic Labs and we released The Light Brigade. This was a VR shooter, which needed to run at 90hz on a Quest 2 (and also released on Steam for PC headsets).
From the store page: "An award-winning roguelike VR shooter with realistic gunplay & immersive light magic."
This game needed to be able to run at 90hz on a Quest 2. As usual, development began on the Steam version (PC) and the team grinded away and getting things working, before making them fast.
The development of the game set out with some clear goals, from pretty early on:
- It needed to be fun.
- It needed to feel larger than life (scale in VR can be huge!)
- Levels needed to be procedural.
- The gunplay had to feel incredible.
We knew that the levels needed to be procedural, in order to ship a title like this from the small team at Funktronic. The prototyping of different procedural generation methods was handled by the company cofounder, Eddie Lee. After a few different methods, he found that Wave Function Collapse (WFC) was the best option for The Light Brigade.
A brief overview of how WFC works:
- Define a lot of 'tiles' (in this case, small chunks of the world).
- Setup connection rules for how any particular tile is allowed or not allowed to connect to every other tile.
- Slot in a START and END tile (start of level, end of level).
- Based on a seed, randomly select neighbor tiles around the start until you flood fill the level until the end, accepting only valid connections.
The end result is that you can get a pretty reasonable looking level, with some simple rules. Funktronic hired a level designer (Eddie Peters) to hand tweak values over and over until the levels started to look almost hand crafted.
The Problem(s)
The levels looked great. Design goals were met: scale was incredible, levels felt immersive, and a player could easily get lost in huge levels. Unfortunately, the levels and general content was too much for the Quest 2 to handle. When we first tried to get this game running on the Quest 2, we were met with frequent Out-of-Memory crashes, single-digit framerates, and WFC-generation times in the tens of minutes. It was rough!
But, all was not lost. I took a deep hard look at what was going on, and came up with a game plan at tackling it. My goal: get this game running at full framerate with NO frame drops, without sacrificing the core values of the game itself, and to keep the game 100% the same mechanically across platforms. I wanted things so similar between versions, you could copy/paste your save file from Quest ot PC and just keep playing (spoiler: this works!).
So, what was actually wrong? When I started profiling, it seemed almost hopeless. Everything was slow! Rendering (GPU at 100%, CPU at 100%, RAM near 100%). Opening the Unity profiler, I was met with a timeline that looked like it stretched into eternity. So, where to start?
For issues like this, I try to tackle the most glaring issues first. Looking at the Unity profiler timeline, the lowest hanging fruit was the time the CPU spent issuing commands to the GPU. The main thread spent well over 50 ms simply issuing commands, stalling everything else in the process. The next largest issue was clearly the time the GPU spent on grinding through those commands. The rest of the timeline was pretty large too, but I figured once we resolved whatever this was, we might be able to take on the rest after.
( The Light Brigade shipped with Unity 2021.3 using a custom fork of URP, for reference. )
Break Downs
With rendering being such a huge part of the CPU profiler timeline, I knew two things must be true:
- We're doing too much.
- We're doing something wrong.
When performance is this bad right out of the gate, it's tempting to start cutting things. However, I found that there were quite a few low hanging fruit all over the project. I'll spare you the boring details, but to cut a long story short there were many instances of double-renderers, LODs were not setup anywhere properly, everything was casting shadows. Grinding through all of this basic stuff, I managed to get performance down to a more reasonable level. We went from a ballpark of ~50 ms on the CPU down to ~20 ms.
To take this a step further, I needed to come up with a way to really reduce the amount of work Unity was doing every frame. Unity does have a magical function: StaticBatchingUtility.Combine() - this method will attempt to combine meshes into a format that allows the engine to draw things more effeciently. In theory, Unity can combine many meshes into a single buffer, then use faster DrawMesh() methods with index offsets into that giant buffer. Unfortunately, using this method actually resulted in slower rendering in the Light Brigade. So, I wrote a custom static batch utility. My reasoning for why Unity's must be slow: they don't take spatial contiguity into consideration, and WFC-nature of this game would heavily benefit from that.
My Static Mesh Baker
Because I was writing one just for this project, I aimed to keep it simple. The idea:
- Slap a
BakeStaticMeshGroupcomponent above a group of static objects, in the transfom hierarchy. - Bake these meshes (either manually via a button on the component, or during Build time).
- Don't have to think about it.
There are definitely other ways of going about this, such as baking at runtime on-device, after a WFC level has been built. But, this needed to run on Quest and I did not want to do anything that would negatively impact load times, at this point. So, with this in mind the BakeStaticMeshGroup was created. How it worked:
- Gather all meshes under the node.
- Do some bookkeeping to track which meshes could be merged with which others, based on Materials.
- Combine the meshes, serialize to disk. Copy materials (so modifications don't carry over). Modify materials if necessary.
- Automatically merge LODs, too.
- Disable Shadows on LOD0, duplicate renderer so LOD0 shadows were actually LOD1.
- Disable the original assets in the scene (so the process could be reverted).
- At build time, delete the original assets in the scene (so they don't clutter the game at runtime).
Some extra goodies that ended up coming out of this system: I found that we did not need full precision of meshes. Everything was pretty tightly contained in small chunks, and the batcher could take advantage of that and aggressively lower precision (vert: float16, normals: float16, tangent: unorm8, color: unorm8, uvs: float16). This resulted in a small bump in GPU performance, and pretty drastic memory reductions.
What else could be merged?
This static mesh baker was a good start, but we also had a problem of drawing too much when it came to enemies. Enemies were mostly humanoid, and they used a system similar to what you'd find in a game with a flexible character creator. They had a whole 'equipment' system similar to what you'd typically see for players, and would have outfits procedurally generated at runtime. For some aspects of this game, this would affect things mechanically. For some aspects, this was purely cosmetic. Either way, it resulted in a LOT of SKINNED mesh renderers. At the time, Unity's animator was not friendly at dealing with this. GPU skinning was not yet batched, so all of these individual skinned computations were being done one draw at a time.
So, I wrote a SkinnedMeshCombiner script, to try and merge things as much as possible. The idea:
- Spawn things as normal, procedurally building an enemy's outfit.
- When it's finished, gather all the meshes and materials, determine what is combinable.
- Then, combine aggressively (and, compress!)
This ended up working great, as well. Enemies went from ~10-15 unique renderers per enemy down to ~1-3 per enemy. As a bonus, the same mesh compression trick from the static batcher worked on the skinned mesh combiner. Unfortunately, I had to keep positions at a higher precision (float32), but everything else could be crunched all the same.
One issue I ran into with combining skinned meshes: the bone list from one mesh to the next was often different. Unity seemed to only import the bones necessary for animation per mesh (or maybe they were exported that way, I'm not 100% sure). Either way, I needed to handle this. I ended up writing a BoneRemapper helper, which was a jobified system to simply iterate over the known bones and re-map them as necessary, before and after a skinned mesh combine pass.
What else could be done?
With both static and dynamic renderers being aggressively merged across the project, and LODs setup properly, double renderers fixed, shadows being LOD biased up by one, performance was still pretty rough, often over ~10ms on the CPU side of things.
Looking at the profiler, I knew something was wrong with how graphics were being drawn on the Quest. The main thread was doing a lot of work, and tons of job threads were sitting empty. This seemed wrong, as we had Graphics Jobs and Multithreaded rendering both enabled in the Project Settings.
Through some trial and error, I found that disabling graphics jobs and enabling multithreaded rendering drastically decreased the work being done on the CPU. I was so shocked by this, I actually asked about it on the Unity forums, assuming it was a bug.
Unfortunately, the answer I was given was a bit disappointing:
Enabling Graphics Jobs would actually implicitly disable LZR/HSR?!?! This was unfortunate, but it was nice to get a confirmation about it. But, you might be wondering, what is he talking about?
Early Z Rejection
The Quest 2 uses a Qualcomm Snapdragon XR2, a mobile chip which handles graphics with a deferred pipeline. Not deferred in the typical graphics-pipeline sense (deferring lights to a separate pass), but deferred in that you submit all your triangles and the hardware will actually try to early-z reject as many of them as possible, to save on fragment shading costs.
The main takeaways with this are:
- Developers should lean into early-z-rejection as much as possible.
- When it's working, the GPUs fragment time should be quite low.
- A lot of stuff can break it, AVOID BREAKING IT.
- Verify things are working via RenderDoc as often as possible.
We knew about this feature early on in development and made a constant conscious effort to avoid breaking it. The number 1 rule was NO TRANSPARENCY (with a few exceptions). If you take a deep look at The Light Brigade, even on the PC version, you'll notice there is very little transparency at all. Everything is mesh particles, lights, or custom fog. The artists (Pat, Kat, Alex) were all extremely good at utilizing various techniques at faking transparency. My personal favorite thing they often did: flip the cull mode on a round object to make it seem like you were looking through it (the fire used this, for layering materials).
Knowing that the hardware pipeline is going to handle rejecting triangles for us, I had a thought: why bother sorting? I disabled Unity's sorting of opaque objects via Camera.opaqueSortMode = OpaqueSortMode.NoDistanceSort - this essentially tells Unity to skip sorting entirely and push straight to the GPU. After doing this, I verified that the CPU time went slightly down, and the GPU time stayed immeasurably the same.
Instanced Stereo Rendering / Multiview
The Quest 2 has two lenses that peer into a single display. However, this display needs to be rendered to twice, one for the left eye and one for the right. By default, in the version of Unity we were using at the time, Unity would simply render everything twice. This is called "Multi-Pass Rendering". It's the easiest to do because it has no extra setup required and 'just works' for basically every game. The obvious tradeoff is: everything takes twice as long to do. You submit everything on the CPU twice, you draw everything on the GPU twice. There's a lot of wasted effort going on.
Now a days, if you are using URP / ShaderGraph, this is typically handled for you. But during the development of The Light Brigade, we had an entirely custom shader library. Thankfully, this wasn't actually too much of a hassle to handle. Unity has a few built in macros that you slap around your shaders, and everything mostly just works.
Implementing Multiview resulted in a near halving of our CPU side costs and a small reduction of in the GPU costs.
Verifying this was working required attaching the Unity Frame Debugger and checking that Unity was rendering into a texture array, instead of a typical texture.
Sampler Counts
One of the major issues with mobile GPUs is their lack of VRAM and small cache sizes. Texture/sampler caches are tiny, so developers need to keep that in mind. Cutting this down in The Light Brigade was done in a few ways:
- Aggressively combine textures, to reduce sample counts.
- Drastically reduce texture sizes, for less texture cache misses.
- Sample extremely small textures in vertex shaders, not just in fragment shaders. Interpolate.
If noise was ever needed, we used a multi-channel noise texture, with very unique settings on each channel. Needed a lot of noise samples? Just read from the R, G, B, and A channels independently. Essentially any use of "texture A is used as an input to sample from texture B" was banned on the Quest port (ruined texture cache hits, constantly stalled fragment shaders). This was done via shader compiler defines (#ifdef / #endif).
The Light Brigade used a typical toon-ramp texture setup, but on the Quest 2 version of the game I moved the ramp texture into the vertex pass rather than the fragment pass. In fact, most of the lighting was handled in the vertex shader on the Quest version. You can see this 'mode' on steam if you lower the shader quality settings to the minimum.
Shader Complexity
Register pressure ended up being a constant headache on Quest 2 version of The Light Brigade. This was mostly a consequence of very complex shaders, in every conceivable metric. They were long, had lots of branches, sampled from many many textures, and were tricky to debug.
One of the first things I did when joining the project was to try and "unify then simplify" the entire shading pipeline. I wrapped everything up in Macros, unified as much code as possible, and implemented custom fog and lighting through those #includes and macros.
When it was all said and done, every shader flowed through the same lighting and fog passes, which were configurable at both global and material levels. I minimized the number of keywords as much as reasonably possible given the timeframe, and removed almost every branch. Not all branches are evil, of course, but many really needed to go.
After going through these low hanging fruit, I tried to find what else could be optimized away. Through profiling, trial-and-error, and good old reasoning, I found that the biggest gains I could achieve would be by hoisting as much as possible OUT of the fragment passes and IN to the vertex passes. This turned out to be relatively easy, but I did eventually run into more performance issues by shoving too much into the vertex interpolators. Profiling with RenderDoc, I found that register pressure was the biggest bottleneck, so converting as many floats into halfs as possible actually gave extremely significant gains across the board, on the GPU.
I also found that doing as much on the CPU and passing it along to the GPU as possible was essentially free gains, anywhere it could feasibly be done.
Application Space-Warp
After all of this was done, we were able to mostly hit 72 fps (base target on Quest 2). BUT! What if we could go further? 72 was OK, but if the game could run at 90 fps it would really feel good. I did spend some more time optimizing shaders but felt that I had essentially hit a wall. Any further gains would have to be large structural changes to how things were developed, and we did not have time for that.
In comes: Application Space-Warp (ASW). The idea:
- Render a motion vector texture. Feed it into the Quest 2's pipeline.
- Quest 2 will generate in-between frames via interpolation.
- You get 1 frame of latency added, but render twice as many frames.
In theory, we could render at 45 fps (!!!) and get a smooth 90 hz output. This sounded too good to be true, and honestly I was extremely skeptical about it. Surely the extra latency would give motion sickness, or I would notice weird artifacting? Well, I tried their demos and was very impressed.
So I thought to myself, why not? I'll spend a day or two on this and see where it goes. At that time in development, you had to download a custom fork of URP in order to generate the motion vectors required for ASW to process. This turned out to be a whole thing, because our render/shading was so customized. I had to go through all of the shaders and use their special macros for generating motion vectors (custom pass), and be very careful about what exactly was being rendered. I also ran into some issues with general motion in the game resulting in full-screen incorrect vectors, but after some fiddling with the URP source I did eventually get this all sorted.
The end result really shocked me: everything basically looked the same, but ran much smoother. What a dream come true? We were optimizing for 72 fps, and suddenly now we only had to hit 45 fps?!?! I don't have much more to say here, Meta really knocked the ball out of the park on this feature, it was incredible to have. It bridged the gap from 72 fps to 90 fps!
That being said, there were many nuances to take into consideration. What to do with (the few) semitransparent objects? What about quickly rotating renderers? What about very fast projectiles? What about teleporting?
For semitransparent things (such as UI cards) I ended up just placing an 'invisible' (motion-vector-only) renderer behind them, which more or less resolved most artifacts around them. I also modified our motion vector macro to clamp rotation and velocity values, to avoid huge artifacts from sudden movements or unnatural rotations. When teleporting, I had the whole system actually disable and then re-enable a frame later. I was worried this would be a noticeable frame drop, but surprisingly turned out to be seamless.
Shader Stutters
Towards the end of development, we had the game relatively stable at 45 fps (90 hz), but would very often get frame drops. After some investigating, these turned out to be typical shader stutters you see in many modern games.
The solution was pretty straightforward: after WFC generation, gather a list of materials and pre-render everything. I also included a small list of 'spawnables' to pre-spawn and pre-render. This effectively reduced almost all frame drops to zero.
Conclusion
The Light Brigade was about a 3 year project, with lots of effort from every developer involved. We managed to get it running very smoothly on the Quest 2, and later added extra graphical options when doing the Quest 3 port. I was very pleased with many of the tools available to us from Unity and from Meta, and especially grateful for RenderDoc. It was a tough journey, but I think everyone involved was greatly satisfied with the result.
If you haven't played the game, please check it out on Steam. It's compatible with most headsets as of the time of writing.