Intended for models that are fairly detailed and contain a large amount of geometric information.
oom occurs when a Chrome tab has more than 4GB of memory allocated. Is there a way to use less memory using indexedDB?
Or has anyone tried another method?
Simplification is difficult to apply.
What exactly is the question here? Which memory? Use for what?
Of course, it is used when rendering. We will use this to render a static model in the scene.
And if a model is too large, it will be subject to constraints contained in the link content, causing oom.
document
For models that take up too much memory, we want a workaround to avoid memory usage. Simplification is impossible. Because even if it is drawn in 3D, it must be viewed flat.
If you ask me what I’m doing this for, all I can say is company work. I just render a huge model without simplification, but oom occurs due to memory usage, so I want to solve this problem.
I don’t know of these limitations off the top of my head, but my reasoning is that an average machine has much more main memory than gpu memory. Not sure how you would draw this if you can’t even interpret a string representing it. Why don’t you break up your models into smaller pieces?
I don’t think IndexedDB is not going to help here. If it’s being drawn, it can’t be only on disk, it must be uploaded to the GPU, and I don’t know if it’s possible to keep resources in memory on the GPU without also keeping them in Chrome’s memory.
Just to check — you’re sure this is mostly geometry? Often textures are a cause of high memory use, they can occupy much more memory than the source images.
You might consider quantization (reduce float32 attributes to uint8 or uint16) to reduce memory overhead. Or if you don’t need to render the entire geometry at the same time, a tiled approach to loading and unloading geometry, such as OGC’s 3D Tiles.
1 Like
Is this OBJ? If your model is reaching maximum string length limits, I would strongly recommend using a more efficient file format.
1 Like
Yes, I’m sure. This is because the dxf file is read and triangulated, so the vertices are all there is.
It is not an OBJ file and is pure text.
1 Like
Well, try binary?
1 Like
Using a text based format can cost more than 10 to 100x more memory than an efficiently packed format like binary gltf.
I basically Never use any format besides gltf nowadays.
Even if you have to convert your DXFs offline on the server somehow to GLTF, that is your best hope at being able to view them in the browser efficiently… and that is what most web based 3d renderers/content generation tools do in some form or another.
2 Likes
I saw this forum and felt inclined to answer because I’m writing a version of IndexedDB that stores data on the video card called VideoDB. The concept grew out of a desire to overcome memory limitations when working with large datasets in web applications, particularly in GPU-intensive tasks like rendering, data processing, or AI workflows. By structuring the data as key-value pairs on the CPU (for metadata) and then placing the actual bytes in GPU buffers, I aim to harness the speed of GPU memory without duplicating everything in the main memory for extended periods.
Currently, my VideoDB class handles tasks such as creating separate “stores,” uploading typed arrays or JSON objects as rows, and retrieving them later. On the CPU side, it keeps a simple table of metadata (row IDs, offsets, etc.), while the GPU is used purely for raw data storage. That arrangement lets me maintain clarity about what data is alive, inactive, or flagged for overwrite. At this stage, the core functionality is stabile. You can create, list, and delete stores, or put and get rows all from the GPU via typed arrays or JSON serialization.
Eventually, I plan to integrate IndexedDB on top of this system to ensure data persistence across sessions. The idea is that the CPU-side metadata, and possibly chunks of GPU data, can be pushed into IndexedDB when not in use. This approach would enable me to restore the data without having to re-upload it each time a user visits the page.
By keeping data in GPU buffers rather than transferring it back and forth between the CPU and GPU for every operation, VideoDB can significantly reduce overhead for AI/ML workloads and other large-scale data processes.
This class is in active development. I don’t even have the code published to GitHub yet, I should have this done by the end of today.
2 Likes
Wait, can you elaborate more? If you refresh the page, what happens with your data?
1 Like
If you refresh your page your data goes away unfortunately. That isn’t the point, you skipped passed all the useful work this can do.
[RESULTS] Stress Test Performance:
jsonStress: ADD=93,925 rec/sec, PUT=9,483 rec/sec, DEL=19,165 rec/sec
float32Stress: ADD=400,877 rec/sec, PUT=9,959 rec/sec, DEL=19,116 rec/sec
float64Stress: ADD=347,247 rec/sec, PUT=9,983 rec/sec, DEL=18,289 rec/sec
int32Stress: ADD=389,708 rec/sec, PUT=9,964 rec/sec, DEL=19,016 rec/sec
uint8Stress: ADD=421,836 rec/sec, PUT=9,987 rec/sec, DEL=18,737 rec/sec
[RESULTS] Total Data Transferred (ADD/PUT): 8.12 GB
Each row is 1kb in size. Can you load 400,877 1kb rows into an IndexedDB database in one second, then access them without ever using your CPU? I have a rather crappy RTX 3090. Yes it has 24GB but it’s slow compared to newer cards.
Putting data on the same device as it’s needed is a game changer. Putting data in an IndexedDB database is slow and wrong because it’s the wrong device.
You can run your own stress test to test your own video card at that link. I’m actively developing this - so expect even more improvements.
1 Like
I’m just confused about why is this compared to indexdb. Indexdb is for storing data on disk if I understand correctly, it’s the complete opposite direction of the gpu.
1 Like
I’m comparing my tool to IndexedDB because it has the same API interface as IndexedDB, and because existing AI based javascript tools use it now.
The performance metrics I posted are remarkably high, especially considering that they were obtained using an older RTX 3090. Inserting nearly half a million 1 KB rows per second indicates substantial throughput, which libraries such as TensorFlow.js, ONNX Runtime Web, and Brain.js could leverage effectively. By using GPU-backed buffers and deferring writes until necessary, these frameworks minimize time spent on data transfers and maximize time devoted to inference or training tasks. For instance, observing ADD rates of approximately 400,000 records per second in the Float32 benchmark suggests that TensorFlow.js could seamlessly handle large-scale video or image datasets, while ONNX Runtime Web could store extensive model weights and selectively retrieve them on demand.
Furthermore, because this class can store data in typed arrays—the same format the video card uses—there is no need for costly JSON serialization or deserialization, leading to lower overhead when dealing with large data volumes. Even Brain.js, despite its more lightweight architecture, would benefit from the immediate transfer of network parameters or training data to the GPU, eliminating the burden of frequent re-uploads.
The benchmark on my site of 8.12 GB of data processed during the tests underscores the advantages of the row-based metadata approach and single mapping strategy, offering a highly efficient solution for rapid, large-volume GPU operations in a JavaScript environment.
3 Likes
My ridiculously stupid question marked the beginning of an incredible project. Truly amazing.
1 Like
I couldn’t disagree more. You don’t realize just how good your question was, and I get that. You should give yourself more credit.
Chat GPT 4o found your post and identified it as “someone who could use my code”. It thought your question was the best one to satisfy my query, and I agree.
2 Likes
I just found out some things about my project that pretty much ended it.
You can’t read data that is stored on the video card fast. It’s extremely slow. Like it could only pull 300k per second, a few people discovered this but I found out the hard way by actually trying it. Maybe that’s okay for some but not for most - and it’s not fast enough to justify further development.
You also can’t share data you push to the video card with other applications. The data pushed to the card via WebGPU is sandboxed into that WebGPU session and can’t be shared or even seen by anything else.
You asked a good question and I thought I could answer it with a novel solution but the design of video cards and WebGPU prevents it unfortunately.
Well as usual it looks like I gave up to soon. This is a viable project again! It turns out my batched get was just calling the code that did a single get instead of making a true batched call.
PUT Performance:
[RESULTS] Stress Test Performance:
jsonStress: ADD=89,958 rec/sec, PUT=8,855 rec/sec, DEL=18,819 rec/sec
float32Stress: ADD=360,697 rec/sec, PUT=9,739 rec/sec, DEL=19,063 rec/sec
float64Stress: ADD=311,022 rec/sec, PUT=9,732 rec/sec, DEL=19,304 rec/sec
int32Stress: ADD=381,666 rec/sec, PUT=9,542 rec/sec, DEL=19,777 rec/sec
uint8Stress: ADD=403,374 rec/sec, PUT=9,740 rec/sec, DEL=18,932 rec/sec
[RESULTS] Total Data Transferred (ADD/PUT): 7.60 GB
GET Performance:
[STEP] Creating 1 store(s), each with 10000 records…
[SUCCESS] All stores created and populated.
[STEP] Verifying accuracy on 1000 random rows (out of 10000)…
[SUCCESS] All sampled records match exactly! Accuracy confirmed.
[STEP] Starting SEQUENTIAL GET performance test (3 seconds)…
[RESULTS] SEQUENTIAL GET: ~259 GETs/sec
[STEP] Starting RANDOM GET performance test (3 seconds)…
[RESULTS] RANDOM GET: ~304 GETs/sec
[STEP] Starting NON-CONTIGUOUS BATCHED GET performance test (3 seconds)…
[RESULTS] NON-CONTIGUOUS BATCHED GET: ~5,636 GETs/sec
[STEP] Starting CONTIGUOUS BATCHED GET performance test (3 seconds)…
[RESULTS] CONTIGUOUS BATCHED GET: ~7,306 GETs/sec
[INFO] Accuracy Test / GET Benchmark Completed.
1 Like
It’s not fair to toy with a man’s heart like that bro