Three-Live & XR-Live: Run procedural Three.js natively on Android Filament & iOS RealityKit (and WebXR)

Showcase
1

I recently open-sourced Three-Live (three-live and xr-live on npm, three-live-compiler on crates.io and SwiftPM) and wanted to share it with the Three.js community!

Why I built it

THREE.js code is super popular, for a reason! You can create a complete scene, materials, and animations in just a few lines of JavaScript. People and LLMs are familiar with writing it.

Where it gets tricky is taking those 3D scenes to mobile:

  • Mobile AR & WebViews: iOS Safari, WKWebView, and Android WebView don’t support WebXR, and exporting a static .usdz or .glb file to OS previewers (AR Quick Look / Scene Viewer) drops onUpdate loops, interactivity, and custom plugins.
  • Native mobile & headset apps: Shipping Three.js inside native Android, iOS, or headset apps usually means either rewriting the scene in Kotlin (Filament / Jetpack XR SceneCore) and Swift (RealityKit), or embedding a WebView that can’t render into the headset’s shared space.
  • Sandboxing dynamic scripts: If a scene script comes from user input or an LLM, running it on the main thread isn’t safe :frowning:

Three-Live is my attempt at a solution.

How it works

Three-Live lets you write standard procedural Three.js and pass the script to native mobile SDKs (or use it on the Web, of course!):

  • On Android (com.google.threelive): Uses an in-process Rust/Boa compiler (three-live-compiler) over zero-copy JNI to compile the Three.js script into binary GLB + KHR_interactivity graphs, driving Google Filament, ARCore, and Jetpack XR SceneCore.
  • On iOS, macOS & visionOS (ThreeLiveCompiler): Compiles the same script into Apple RealityKit Entity hierarchies (RealityKitHelper), ARView / RealityView, and Metal.
  • On the Web: Evaluates the script inside a sandboxed Web Worker or opaque-origin iframe (connect-src 'none', CPU time and geometry quotas), streams the validated scene graph to a lightweight WebGL2/WebXR renderer, and includes in-memory .glb and .usdz packagers for 1-tap mobile AR handoff.

Quick example (Android)

import com.google.threelive.FilamentHelper
import com.google.threelive.XRLiveRuntimeSession
import com.google.threelive.XRLiveSceneCoreHelper

// Compile a Three.js script to a zero-copy GLB buffer and load in Google Filament
val holder = FilamentHelper.loadModelZeroCopy(script) { buffer ->
    assetLoader.createAssetFromBinary(buffer)
}

// Drive KHR_interactivity graphs, head-follow springs, and spatial/sim plugins each frame
val runtime = XRLiveRuntimeSession.fromGlb(holder.byteBuffer)
runtime.step(dtSeconds = 1.0 / 60.0)

// Or attach interactive spatial traits (grab, scale, billboard, anchor) in Android XR SceneCore
XRLiveSceneCoreHelper.attachTraitsFromGlb(xrSession, entity, holder.byteBuffer)

Example (Web)

Modular plugin architecture

To keep the core web bundles small and dependency-free, advanced features are split into 28 optional plugins that have matching implementations in Android, iOS and Web runtimes:

  • Core 3D & Animation (<three-live>): glTF KHR_interactivity behavior graphs, PBR Next materials (clearcoat, transmission, sheen, iridescence, anisotropy), HDRI & spherical-harmonics lighting, contact shadows, 3D surface hotspots, 64-joint GPU skinning, morph targets, and keyframe cross-fading.
  • Spatial Perception, Splats & AI (<xr-live>): lazy head/hand follow, 3D manipulation handles, plane/mesh world understanding & 2D-to-3D object detection, real-time Gemini Live vision/audio streaming, 3D Catmull-Rom spatial paint, depth-colliding particles, and .splat / .spz Gaussian splats.
  • CAD Review, Layout, Locomotion & Co-presence (<xr-live>): live bounding-box dimension callouts & AR tape measure, magnetic socket/grid snapping, eye/head gaze dwell, multi-part CAD exploded views & GPU cross-section plane slicing, persistent anchors (with shareable #xr-state URL hashes), teleport/snap-turn locomotion, and WebSocket multi-user co-presence.
  • Simulation, Agents & Scenarios (xr-live-sim): capsule character controller, autonomous steering agents (patrol, chase, flee, follow, wander), hit/integrity/trigger zones & template spawners, checklist objectives with timers/scoring, and zero-asset procedural Web Audio / native audio synthesis.

Full disclosure: I built this as a solo project at Google with Antigravity. I’d love for folks to try out the playgrounds and let me know what works well or what’s missing!

Jonathan