Built-in viewer is self-contained so very little code is necessary to load and view a scene
Allows user to import .ply files for conversion to custom compressed .splat file format
Allows a Three.js scene or object group to be rendered along with the splats
Known issues
Splat sort runs on the CPU – would be great to figure out a GPU-based approach
Artifacts are visible when you move or rotate too fast (due to CPU-based splat sort)
Sub-optimal performance on mobile devices
Custom .splat file format still needs work, especially around compression
It is still very much a work in progress at this point, but I wanted to share what I’ve been working on for the past month or so. I would also love feedback, critiques, suggestions, or any other thoughts that would help me make it better!
Looks gorgeous, thanks for sharing! One thing I’d love to see is a progressive loading feature, so we don’t have to wait for the entire .splat file (often in the hundreds MB). I’m not sure about the technical details, but maybe reading chunks of data while loading the file (ReadableStream? or this SO!) or splitting the file into multiple parts could be an option!
Poly.cam from the network console seems to be implementing the first option.
Thanks! Progressive loading is definitely something I’m planning on adding eventually. I have already implemented an incremental loading function fetchWithProgress() , and it would be straightforward enough to build off of that. I think the key will be organizing the data in the .splat files in a way that yields a nice visual result as it’s loaded and is simultaneously conducive to compression.
The project looks beyond awesome, but basic information is lacking.
Like a simple description of the concept - I had to download a 115MB(!) paper to find out that it’s a method of 3D reconstruction out of images.
Also, is this a viewer of pre-made scenes?
How can one make his/her own by taking pictures of a scene?
If the 3D scenes are reconstructed from images what is the loading of a ply file for?
Is it possible to load the 650MB scenes file from the offiicial page?
… Like a simple description of the concept - I had to download a 115MB(!) paper to find out that it’s a method of 3D reconstruction out of images.
I admit that my current description is somewhat vague regarding it’s purpose and relationship to the original work by the folks at INRIA. My project is merely a viewer/renderer for the scenes (stored in .ply files) that can be generated by their software. I should note that they refer to scenes as “pre-trained models”.
How can one make his/her own by taking pictures of a scene?
If you want to construct your own scene from images, you can take a stab at using their software, but companies like Poly.cam and Luma AI have online services for exactly that purpose.
If the 3D scenes are reconstructed from images what is the loading of a ply file for?
The .ply files contain the scene that is generated from the images. The data in the file is similar to a point cloud.
Is it possible to load the 650MB scenes file from the official page?
Yes, but you’ll actually want to download their archive of pre-trained models, which contain the .ply files: INRIA pre-trained models
Wow! An easy to use online Gaussian Plat viewer! Great job!
Your controls are perfect for online viewing of splat files; both desktop and mobile (iPhone). I also like that you placed camera controls to control initial viewing. (I did notice that I had to change default Camera Up values to 0,-1,0 for proper orientation.)
My goal is to find ways to embed gaussian splats on my website quickly and easily. What’s cool is everything is headed in that direction. I was able to find someone using a-frame code inside iframe to embed splats and it actually works pretty good but still seems somewhat limiting.
@Skia
WebGPU has zero mobile support, and only Chomium-based browsers support it on PC- I don’t think it is required anyway, Webgl-2 shaders most likely can do the job.
@mkkellogg This is really really good repo, thank you!
How much headroom do you think it can be optimized further for better performance on mobile?
Can you list some viable route for further optimization that you thought of, that’d be greatly appreciated and we can contribute in some way if possible.
@dllb
It’s quite normal for new features to be chrome first (and firefox behind a flag) and then everything else, so hopefully mobile will have support soon. Porting the code would be trivial.
Nonetheless you are right, everyone favourite solution would be to webgl 1/2.
I guess the alternative is to “render” the sort to a texture, where the bytes of this textures are the index order and frequencies.
Then pass to the current vertex shader the texture and the whole geometry attributes as a uniform and use the texture as a lookup.
(So basically keep the same indexes, but change what they refer to)
The biggest reason for performance issues on mobile devices (based on what I’ve observed) is that they simply struggle to render large numbers of splats every frame. I think the optimization path most likely to yield substantial results would be investigating ways to reduce the number of splats that have to be rendered.
I have been thinking about how an LOD mechanism of some sort might help with this goal. The trick would be to make sections of the scene that are far away appear the same using fewer splats. But you can’t just use a basic heuristic (like only render the top N when ordered by opacity) to decide which splats to skip because it just won’t look right.
If you can figure out a nice way to do that, I’d be happy to include it
I’ve thought a lot about how I could use “traditional” rasterization techniques (using standard vertex & fragment shaders along with vertex based geometry) to sort the splats and I just can’t figure out a nice way to do it. Not only that, but using the GPU for sorting that way would take resources away from rendering.
I think I’m going to need to wait for WebGPU to be more widely supported before I start experimenting with sorting on the GPU. Of course I’d be happy to be proven wrong… but all other web-based (non-WebGPU) approaches that I’ve seen also use the CPU for sorting.
this is looking very good! i wonder, is it planned to make it more generic? as in less of an engine and more of a package/re-usable? it would be super useful to mix splats with common threejs scenes. it’s possible according to the docs but not without taking over threejs and injecting an existing scene.
This is definitely something I have thought about, and I know there are several developers who would like the ability to just drop splat scenes into a three.js scene. I just haven’t figured out a good way to do it. One of the biggest hurdles is managing the sorting of the splats. If I want to support multiple splat scenes within a single Three.js scene, their splats have to all be sorted and rendered together, which would require some sort of external mechanism that links them, merges their splat data, and can schedule sorting based on camera movement (and that’s already starting to sound like an engine). If we want to simplify the problem and support only a single splat scene at a time, we still have the issue of scheduling and optimizing sorting (in this case via a custom octree) and managing a separate worker thread for it. It all seems fairly heavy weight for a renderable Three.js object.
That is not to say it isn’t possible to accomplish what you’re suggesting – I just don’t think it’s going to be super straight-forward. I’d be happy to entertain ideas on how to improve the Three.js integration.
it would need some sort of higher order composition. but perhaps some lower level vanilla set up could be built out for this so that higher order systems can componetize it. i would be very interested in building react/drei components for this. in that world composition is a given and it would fit in beautifully. we have countless of components that do just this, for instance clouds GitHub - pmndrs/drei: 🥉 useful helpers for react-three-fiber where it also has to accumulate instances and figure out sorting.
would you want to consider building it out with us with Poimandres · GitHub ? pmndrs is for a dev collective for threejs, we maintain a couple of vanilla projects (postprocessing, cannon, p2, meshline, …) and lots of devs willing to dig in and help. it would stay vanilla, we help you promote and figure out lower level patterns, we build the higher level layer, and it’ll reach countless of people that way.
I’m not aware of the specific algorithm for this purpose, but since we can write WebGL-1/2 shaders that can read data from one or more textures as input and write to one or more textures as output with multiple passes if needed, I don’t see why we can’t ‘compute’ anything given.
Is that sorting always required? How about converting those weird formats to a regular Three.js scene and avoid all sorting, and screen-refreshing lag too?
A composition approach makes a lot of sense; to that end it might be a better idea to have Viewer act as both a container for one or more instances of SplatMesh, and a renderable object that can be added to a Three.js scene. It could still properly manage splat sorting for its child instances of SplatMesh by implementing a callback for onBeforeRender(), and there would be no need to manually call Viewer.update() or Viewer.render(). The result would be a “renderable” that is still fairly heavy-weight … maybe that’s OK?
I’m happy to consider collaborating with the Poimandres folks if this is something people will find useful.
it would also mean that some bits could come out from the renderable viewer itself, ie the orbit controller could be an import.
it’s fairly heavy, but that’s because of the nature of the splats, so it doesn’t really matter if you are adding 3D objects to viewer.scene or adding the renderable viewer to the scene, but having it as a renderable would be more in line with the general threejs approach i think.
At that point the worker can just be another transform feedback
