A live 3D digital twin of my house in three.js, driven by Home Assistant (procedural materials, raymarched smoke, in-browser 4K video export)

Showcase
1

Hi everyone! This is my house as a live 3D replica in the browser. It’s wired to Home Assistant (the open-source home automation platform), so everything in it reflects the real house right now: lights, awnings, the garage door, smoke and leak sensors, the rain and wind from my weather station, even where each of us is. You can also control the house from it.

There’s no public demo, since it’s connected to my real home, but both videos were rendered by the app itself, frame by frame, in the browser.

Architectural plan look
Scale model look
Ink drawing look
Paper and cardboard look

From floor plans to GLB

  • The house isn’t modeled by hand: Python scripts build the Blender file from scratch out of the house data (taken from the original floor plans and photos), with metadata on every object: what it is, its floor, its device.
  • It’s exported as a Draco-compressed GLB, one node per object, with the animations made in Blender (a door opening, an awning extending). An AnimationMixer plays them according to the real state, so an awning at 40% is drawn at 40%.
  • Materials are rebuilt from their Blender definitions. Wood, tiles, marble and fabrics are procedural, computed in the shader from world position like in Blender, through onBeforeCompile.

Lighting and post-processing

  • The sun is where it really is, from the time of day and the real orientation of the house. Its VSM shadow map is only re-rendered when the sun moves.
  • Lamps share a fixed pool of spotlights, assigned to the most relevant visible lamps, so shaders never recompile when you switch floors.
  • EffectComposer with GTAO, bloom, SMAA and outlines, and AgX with a color transform that matches Blender’s compositor, so the web and the Blender renders look the same.
  • Emissive strips don’t light anything in a rasterizer, so a halo pass paints the glow they would leave on the wall behind them.
  • A soft contact shadow under the house: it’s rendered from below, blurred and laid on a plane.

Effects

  • Rain: drawn on top of the finished frame using the house’s depth. While only the rain moves, each frame is the cached house plus one draw call of GPU-animated drops.
  • Smoke: when a smoke detector goes off, its room fills with raymarched smoke: 3D noise in three drifting layers inside the room’s volume, lit by the house lights.
  • Room cutaways: walls and slabs are clipped per pixel against the room’s outline in the shader, and the open cuts are capped with a flat section color that the AO also sees as flat.
  • Looks: the same scene as a scale model, an ink drawing (from the AO pass’s normals and depth, so no triangles show), paper, clay, a color illustration, an X-ray wireframe and an architectural plan with dimensions. They switch with a sweep of light.
  • Moving between floors cross-fades two renders with the current camera on every frame.

Performance

  • It only renders when something changes: a still house draws nothing.
  • While the camera moves it skips AO and freezes the sun shadows, and full quality comes back as soon as it stops.
  • Picking uses three-mesh-bvh. It holds 60 fps on an iPhone.

Video export in the browser

The demo editor renders MP4s up to 4K at 60 fps without screen recording. A virtual clock takes over performance.now, Date.now, timers and requestAnimationFrame, so time advances exactly one frame at a time, however long each frame takes. Each frame is encoded to H.264 with WebCodecs through mediabunny, and the HTML overlays (panels, labels, tooltips) are rasterized into the frame through SVG foreignObject.

And it knows where we are

People appear as photo discs at the X/Y/Z position that my own Bluetooth positioning engine computes from 21 proxies. With the proxies shown, each one emits waves, and dashed lines link each person to the proxies that weigh most in locating them.

Write-ups

Stack: three.js r186, Vite, a small Node.js gateway to Home Assistant’s WebSocket API, and Playwright tests in a real browser. I built it in about two weeks with AI coding agents (GitHub Copilot). The code isn’t public yet: the part that generates the house is still very specific to mine.

Happy to go deeper into any of the techniques!

1 Like