i’ve tried porting the antimatter impl which was also used by aframe, this is how a renderable would behave:
so everything is now under the users control, the type of camera, the scene, controls. but it’s lacking unified instancing for more than 1 splat. it would be silly easy from reacts perspective to manage that, but the shader stuff and the worker is quite overwhelming, it looks like the antimatter codebase hasn’t prepared for that yet.
Thanks for the example! I think making my code work in a similar fashion will be pretty easy, but supporting multiple splat scenes is going to be the tricky part. Is it worth releasing a drop-in version or R3f version that only supports a single splat scene?
i think i will attempt managing multiple splats today, i guess it’s just about pumping bufferarrays into the worker. though removing splats is probably going to be hard. i would suggest you keep it vanilla as extensible functions. instead of an engine people will have to create a splatmanager class and interface the renderable object3d’s with it by constructor injection or something like that?
i can componetize it into fiber for you if you want a broader audience.
btw, i have never really tried this before because i thought this needs specialized hardware. but i studied a bit and turns out it’s actually super easy to create. this really is a game changer, i don’t have to be a blender artist to make showcases on the web.
@mkkellogg i have a very rough draft of multi splat with depth ordering but now i face troubles with position/scale/rotation of the individual splats https://codesandbox.io/s/nostalgic-northcutt-qp4jmf?file=/src/splat/index.js:10329-10710
the original code uses this code to position the splats
let center = new THREE.Vector3(f_buffer[8 * i + 0], f_buffer[8 * i + 1], -f_buffer[8 * i + 2])
let scale = new THREE.Vector3(f_buffer[8 * i + 3 + 0], f_buffer[8 * i + 3 + 1], f_buffer[8 * i + 3 + 2])
let mtx = new THREE.Matrix4()
mtx.makeRotationFromQuaternion(quat)
mtx.transpose()
mtx.scale(scale)
let mtx_t = mtx.clone()
mtx.transpose()
mtx.premultiply(mtx_t)
mtx.setPosition(center)
each splat is a virtual placeholder (a THREE.Group with nothing in it), the world matrix of it is obj.current.matrixWorld, do you know how i can apply it? i tried multiply, premultiply, add, but the results are rather crazy.
I played around with it a little bit and the main problem might be how you’re transforming the 3D covariance matrix, which is calculated with this code:
let mtx = new THREE.Matrix4()
mtx.makeRotationFromQuaternion(quat)
mtx.transpose()
mtx.scale(scale)
let mtx_t = mtx.clone()
mtx.transpose()
mtx.premultiply(mtx_t)
To properly transform a covariance matrix C by matrix M you need to use this formula:
C' = M * C * Mt
where Mt is the transpose of M. You’ll want to transform the covariance separately from the position, and then add the transformed position into mtx
FWIW I have a WIP branch that I’m building on to ultimately create a drop-in capable viewer: WIP: Drop-in/renderable viewer by mkkellogg · Pull Request #60 · mkkellogg/GaussianSplats3D · GitHub. Basically I just wrapped the existing Viewer class in another class that inherits from THREE.Group: RenderableViewer. I also created an example of its usage: Drop-in example. I’ve never worked with R3F before, but I hope it would be simple enough to make an R3F component from that?
The main thing I still need to add is the ability to support multiple splat scenes. My current idea is to just merge new splat scenes into the existing one.
Looks very good, this will make so many threejs users happy, thanks for investing the extra time and thank you for the math hint as well, will try it first thing in the morning.
Btw, what are the main differences between your implementation and the antimatter impl?
Well my code is separated into multiple files and multiple classes, so my hope is that it is easier to read and understand… however I could be totally wrong about that 
But I have also taken additional steps for performance optimization. I use an octree to cull splats that are not visible or near the frustum in order to speed up both rendering and sorting. I implemented my sorting algorithm in C++ and it’s imported via a WASM module. The sort itself uses web assembly SIMD instructions and is currently 100% integer math based (which may become an issue with larger scenes and the potential of overflow). I’m also using a transform feedback to calculate splat distances from the camera prior to sorting. I have also tried to document the shader code so that it is easier to understand how the covariances for each splat are projected and ultimately rasterized. Additionally I created a custom .splat file format with some basic compression that significantly reduces file size, although work on that is ongoing: Universal format discussion
I have to give Kevin Kwok credit though; he implemented his version first and it was a great starting point for mine. I have tried to get mine to where it is 100% my own code… I think I have been mostly successful
My “drop-in” branch is now nearly complete, it supports adding a renderable splat viewer to a three.js scene just like any other three.js renderable. It also supports loading multiple splat scenes and applying custom transformations to those scenes. Example usage can be seen here: Drop-in example. Let me know if this is heading in the right direction
That’s awesome! I’ve been attempting to solve another issue, moving splats and re-using data and worker for multiple splats of the same source. Collected some of the results here GitHub - drcmda/splats it’s live here https://splats.vercel.app/
Maybe the structure can help you, i tried to divide it into a vanilla three loader and material. a dataset that’s loaded once can now be used numerous times. it’s still based on antimatter, i would love to migrate the actual component that’s going to abstract it towards your wasm implementation.
thanks for this i have a dpi8 handheld 3d scanner which generates dense pointclouds im hoping the fidelity is better and that this demo will work in converting these dpi8 .ply files to gsplats thanks for this really appreciated was waiting for someone clever to take the reigns on .ply to gsplat conversion
seems to have trouble importing a 550mb .ply that is exported straight from dot product dpi8 also has trouble loading the ksplat converted.ply i am on a gtx 1060 3gb card hough will report back when tested on rtx 3060 machine 12gb vram
Feel free to open an issue in the Github repo and we can take it from there. I’ve only tested my code with .ply files that are generated specifically for gaussian splatting scenes, so I’m guessing the issue has to do with the fact that you’re trying to use a “standard” point cloud.