Gauzilla: a 3D Gaussian Splatting renderer written in Rust for WebAssembly with lock-free multithreading

Overview

Gauzilla

A 3D Gaussian Splatting (3DGS) renderer written in Rust for platform-agnostic WebAssembly (WASM) with lock-free multithreading.

  • Uses WebGL and CPU splat sorting (based on splat) for high compatibility among web browsers
  • Circumvents WASM's limitations in multithreading via the use of the lock-free bus mechanism
  • Uses rfd to securely load PLY files that are stored locally on the host machine

Screenshot #1

Screenshot #2

Introduction

The innovative rendering technique known as 3D Gaussian Splatting (3DGS) [1] represents a Machine Learning-oriented approach to 3D rendering, specifically designed for Novel View Synthesis (NVS). It facilitates the real-time photorealistic rendering of scenes reconstructed from images and videos captured using conventional smartphone cameras. Since its release in 2023 there has been a Cambrian explosion of 3DGS applications and extensions, such as 4DGS [2], D3GA [3], SLAM [4], SC-GS [5], GPS [6], GHA [7], etc.

One of the original ideas of splatting Gaussian ellipses can be traced back in 2002 [8] and 3DGS uses almost the same approach for its efficient forward rendering on the GPU.

Briefly, a 3D elliptical Gaussian centered at a point p with a covariance matirx V is defined as:

Eq.19 of [8]

The key to splatting it onto the 2D screen is to compute the 3x3 covariance matrix in ray coordinates:

Eq.31 of [8]

where W is the view transformation (camera) matrix, V_k = R * S * S^T * R^T (covariance matrix factorized with a rotation matrix R and a scaling matrix S, both of which can be computed, respectively, from quaternions and scalers stored in the PLY file), and

Eq.29 of [8]

is a Jacobian or the partial derivative of the camera-ray mapping at the point t_k in camera space (i.e., affine approxmation of the perspective projection).

Once the covariance matrix in ray coordinates is obtained via the simple matrix multiplications above, one can compute from the top-left 2x2 component of the matrix the major and minor axes representing the size and orientation of Gaussian ellipses, which can subsequently be rendered on the GPU efficiently using Geometry Instancing.

How to Build & Use

  1. Install the nightly version of Rust: rustup toolchain install nightly
  2. Install wasm-pack and sfz
  3. Run ./build.sh and open the locally-served URL in a web browser
  4. Open a PLY file formatted for 3DGS (eg. download the official pre-trained models)

Orbit Camera Controls:

Left mouse button   - Rotate view around target
Middle mouse button - Zoom in/out
Right mouse button  - Move left/right/up/down

Fly Camera Controls:

Left mouse button   - Change view direction (free-look)
Middle mouse button - Move forward/backward
Right mouse button  - Move left/right/up/down

ToDo

  • Optimize Scene::sort() and Scene::generate_texture() (eg. parallelize using wasm-bindgen-rayon)
  • Display a progress bar for PLY file loading
  • Implement progressive PLY file loading for web hosting
  • Allow camera controls with keyboard
  • Write a WebGPU render path (cf. splatter)

References

  1. Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering," 2023.
  2. Wu et al., "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering," 2023.
  3. Zielonka et al., "D3GA - Drivable 3D Gaussian Avatars," 2023.
  4. Matsuki et al., "Gaussian Splatting SLAM," 2023.
  5. Huang et al., "SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes," 2023.
  6. Zheng et al., "GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis," 2023.
  7. Xu et al., "Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians," 2022.
  8. Zwicker et al., "EWA Splatting," 2002.
You might also like...
This automatically patches the RoPro extension for you, allowing you to have pro_tier for free.

RoPro Patcher This automatically patches the RoPro extension for you, allowing you to have pro_tier for free. NOTE Chrome, Brave (and possibly other b

a FREE and MODERN split-screen tetris game WITHOUT ADS
a FREE and MODERN split-screen tetris game WITHOUT ADS

tetr:: A โœจ modern โœจ Tetris game made in OpenGL and Rust Gameplay tetr:: is an implementaion of modern Tetris, and essentially a clone of tetr.io. This

Desktop app for reading and downloading manga. With clean distraction-free design and no clutter

Tonbun Tonbun is a desktop app for reading and downloading manga. With clean distraction-free design and no clutter. Build with Rust, Tauri, Vue.js, a

A free and open-source DNA Sequencing/Visualization software for bioinformatics research.
A free and open-source DNA Sequencing/Visualization software for bioinformatics research.

DNArchery ๐Ÿงฌ A free and open-source cross-platform DNA Sequencing/Visualization Software for bioinformatics research. A toolkit for instantly performi

A little tool to create region-free openingTitle.arc files for New Super Mario Bros. Wii, or to convert them from one region to another

smallworld ...though the mountains divide and the oceans are wide... smallworld is a little tool that can create region-free openingTitle.arc files fo

An interruption-free notification system for Linux

nofi A Rofi-driven notification manager rofi.mp4 nofi is a distraction-free notification center. While most notification daemons make immediate popups

Command Line Scientific Calculator. Free Forever. Made with โค๏ธ using ๐Ÿฆ€

โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•— โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•— โ–ˆโ–ˆโ•”โ•โ•โ•โ•โ•โ–ˆโ–ˆโ•”โ•โ•โ•โ•โ•โ–ˆโ–ˆโ•”โ•โ•โ•โ•โ• โ–ˆโ–ˆโ•‘ โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ•‘ โ–ˆโ–ˆโ•‘ โ•šโ•โ•โ•โ•โ–ˆโ–ˆโ•‘โ–ˆโ–ˆโ•‘ โ•šโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•—โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•‘โ•šโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ•— โ•šโ•โ•โ•โ•โ•โ•โ•šโ•โ•โ•โ•โ•โ•โ• โ•šโ•โ•โ•โ•โ•โ• -

Free and open public transport routing.

Transitous Free and open public transport routing. Goal A community-run provider-neutral international public transport routing service. Using openly

๐Ÿš+๐Ÿฆž Ultra-portable Rust game engine suited for offline 2D games powered by WebAssembly

pagurus ๐Ÿš + ๐Ÿฆž Ultra-portable Rust game engine suited for offline 2D games powered by WebAssembly. Examples Snake Traditional snake game: examples/sn

Owner
Yoshi Sato
In geometric algebra (GA), a k-blade is any object that can be expressed as the outer product of k vectors, and is of grade k.
Yoshi Sato
Provide free GPT-3.5 API service by reverse engineering the login-free ChatGPT website.

ChatGPT Free API Provide free GPT-3.5 API service by reverse engineering the login-free ChatGPT website. Note: This service requires the IP to be able

null 21 May 5, 2024
Just a simple object renderer, written in under 500 lines using Rust.

All cargoes that the project runs are: bitflags: a crate for defining bitflag types cfg-if: a small macro crate for defining cfg-based - conditional c

null 3 May 4, 2023
A tiny software renderer written in Rust.

tiny-renderer ็ฎ€ๅ•็š„่ฝฏๅ…‰ๆ …ๅŒ–ๆธฒๆŸ“ๅ™จ ๅŸบ็ก€ๆ•ฐๅญฆๅบ“ Bresenham็”ป็บฟ็ฎ—ๆณ• Cohen-Sutherland็บฟๆฎต่ฃๅ‰ช็ฎ—ๆณ• glTFๆจกๅž‹ๅŠ ่ฝฝ ๆจกๅž‹/่ง†ๅ›พ/ๆŠ•ๅฝฑๅ˜ๆข ่ƒŒ้ขๅ‰”้™ค ่ง†ๆคŽๅ‰”้™ค ้ฝๆฌก็ฉบ้—ด่ฃๅ‰ช ๆทฑๅบฆๆต‹่ฏ• Blinnโ€“Phong็€่‰ฒๆจกๅž‹ ้‡ๅฟƒๅๆ ‡ๆ’ๅ€ผ ่ฟ่กŒ bresenham็”ป็บฟ็ฎ—ๆณ•

Night's Watch Games 9 May 29, 2023
A Rust CLI to provide last publish dates for packages in a package-lock.json file

NPM Package Age A Rust CLI which if you provide a npm lockfile (package-lock.json to start), it will give you a listing of all of the packages & the l

Benjamin Lannon 1 Feb 4, 2022
A ranger-like flake.lock viewer [maintainer=@figsoda]

nix-melt A ranger-like flake.lock viewer Usage nix run github:nix-community/nix-melt Usage: nix-melt [OPTIONS] [PATH] Arguments: [PATH] Path to th

Nix community projects 120 Jun 28, 2023
A text renderer for Rust's embedded-graphics crate, based on U8g2

u8g2-fonts This crate is a pure Rust reimplementation of the font subsystem of U8g2. It is intended for the embedded-graphics ecosystem. Licensing Whi

null 4 Aug 29, 2022
A Rust-based renderer for Remotion videos.

Rustymotion A Rust-based renderer for Remotion projects. ?? WARNING: This is exploratory work only, use at your own risk. Design Remotion allows you t

Siddharth 12 Oct 6, 2023
A more free Rust-Iterator.

CURSOR A more free Rust-Iterator. | Examples | Docs | Latest Note | [dependencies] cursor = "1.0.0" Example use cursor::*; fn example7() -> u8 {

Doha Lee 4 Jun 1, 2022
Accompanying code for my talk "No free lunch: Limits of Wasm as a bridge from Rust to JS" presented @ EuroRust2022 in Berlin

No free lunch: Limits of Wasm as a bridge from Rust to JS Accompanying code for the talk I presented at EuroRust 2022 in Berlin, Germany Slides for th

Alberto Schiabel 11 Dec 30, 2022
A simple Rust library for OpenAI API, free from complex async operations and redundant dependencies.

OpenAI API for Rust A community-maintained library provides a simple and convenient way to interact with the OpenAI API. No complex async and redundan

null 6 Apr 4, 2023