We want tonemapping to be able to handle HDR. We could do the tonemapping inline at the end of regular shaders, but this would mean that we couldn't do HDR post processing such as bloom, as well as other effects like accurate transmissive rendering.

This does cause some problems for AR, because by default we're blitting the entire HDR framebuffer to the output framebuffer, which writes over the camera image. I'm not sure if it's possible to copy the camera image to the HDR framebuffer first (need to check) but a better solution is to probably use stencil testing for this so we only blit pixels we've written to.

Currently lighting is just a single fixed directional light and things look pretty ugly. A much better solution would be to load a HDR image (per world) for use as a skybox and IBL.

Babylon.js uses it's own funky format for IBL environment maps: https://doc.babylonjs.com/divingDeeper/materials/using/HDREnvironment#what-is-a-env-tech-deep-dive. It might be worth thinking about using this, but I don't like that it uses packed .PNGs but then treats them as being HDR even when they're not, really.

https://github.com/derkreature/IBLBaker is another project to look into. I think it exports DDSes though but I imagine there's some tool to losslessly convert them to KTX 2s.

Mateversum currently isnt too performant on mobile GPUs such as the Oculus quest 2 and OnePlus Nord (my phone).

A decent amount of this can be attributed to the sampling cost of the cubemaps (can be mitigated by reducting them down to 512x512).

Another source is that we're rendering to a fp16 hdr framebuffer even if we're just immediately tonemapping it. Doing this is useful because it gives us a place to insert effects like bloom, but that's only really viable for discrete GPUs. So we should do the following:

• On high perf GPUs, keep things as is.
• on low perf GPUs when doing VR, tonemap in fragment shaders to the multiview framebuffer, then copy that over to the device framebuffer.
• on low perf GPUs when not doing VR, tonemap in the fragment shader to the device framebuffer

This also gives us the advantage of having a better AR mode, as we would be able to render ontop of the camera view again!

We have basic android chrome AR movement where you hold press in a direction to move in, so we should add something around the same level of effort for VR.

https://gltf-viewer.donmccurdy.com/ displays the Alicia.vrm VRM sample as like this:

while we currently render Alicia.VRM like this:

This is because we're doing tonemapping for unlit materials but probably shouldn't be.

This is very solveable when doing inline tonemapping - just don't tonemap and just do the linear-to-srgb conversion instead! But when we're doing a seperate tonemapping pass it gets harder. How do we tell that a fragment should be tonemapped or not? Perhaps we could write to the alpha channel and tonemap if the alpha is 1.0 and not if it's 0.0.

https://github.com/expenses/mateversum/blob/master/src/lib.rs#L51

Problem: web worker main thread - different global object - not window.foo, but rather worker.foo

To solve: should just be as easy as passing the query param into console_log::init_with_level ?

From https://developer.mozilla.org/en-US/docs/Web/API/WorkerGlobalScope/location

console.log(location); inside a worker should return an object->href shaped like http://localhost:8000/worker.js, which might not contain the query params passed by the user.

I want to look into implementing the XRFoundation's volumetric video codec for Mateversum: https://github.com/XRFoundation/Universal-Volumetric The have demo files here: https://github.com/XRFoundation/UniversalVolumetric-Demo

Parsing the json manifest and loading the geometry into buffers seems easy enough. The harder part is handling the video texture. I made some rough changes to wgpu to allow for external textures a while ago, so I'd need to dig those out.

Just for when I try things out on a quest 2. All of these things should probably be toggleable though.

• lower/off anisotropic filtering
• lower texture sizes.
• change log level (large potential gains here!)
• reduce cubemap sizes to 512x512.
• replace diffuse cubemap with a constant. Ideally use sphere harmonics
• turn off mirror
• test doing a depth pre-pass

Related to https://github.com/expenses/mateversum/issues/28.

I was trying out VR with a headset connected to a poertful gpu and noticed that the flickering from aliasing was quite bad. The regular, hardware based approach to anti aliasing is to use multisampled anti aliasing. Unfortunately we can only multisample (in WebGL) if:

• we're rendering objects directly to the device framebuffer
• OR if we're the Oculus webgl multisampling extension: https://developer.oculus.com/documentation/web/web-multiview/#using-multiview-in-webgl-20 (only implemented in the Oculus browser).

This unfortunately leaves out GPUs that are rendering to a HDR framebuffer for things like bloom. A solution here might be to use a shader-based anti aliasing method like fxaa (or txaa?)

Additionally it's worth mentioning that resolving a multisampled hdr framebuffer has some requirements to allow for better edges. See https://mynameismjp.wordpress.com/2012/10/24/msaa-overview/. Also discussed for bevy: https://github.com/bevyengine/bevy/pull/3425.