Rendering tilemap lod with plan THREE.PlaneGeometry

Hello,

In order to understanding how is working a LoD Image viewer (like openseaDragon for exemple)I’m trying to reproduce this kind of mecanism but more three.js oriented.
So basicaly, i’d like to render texture in webgl instead of canvas (even if i use canvas mecanism to read and crop image).

My LodImage is composed of an array of LevelInfo (depending of zoom level) wich contains an array of lstTile (wich are image’crops ).

class LodImage {
constructor(scene, nodeParent, nameNode) {

                            //Image properties 
                            this._url = '';
                            this._width = 0;
                            this._height = 0;
                            this._tileSize = 0;
                            this._lstLevelInfo = [];

                            //three.js  properties 
                            this._scene = scene;
                            this._graphSceneNodeParent = nodeParent;
                            this._nodeSceneName = nameNode;
                            this._graphSceneNode = null;
                    }

async buildPyramid(imageUrl, tileSize, nameNode) {

                            //compute all resize global images for each level
                            await buildPyramidModel(this, imageUrl, tileSize);


                           createGeometryLevelsTile(this);

                            
                    }

}

class LevelsInfos {
constructor(levelIndex, width, height, RowTiles, ColumnTiles) {

                            this._levelIndex = levelIndex;
                            this._width = width;
                            this._height = height;
                            this._rowTiles = RowTiles;
                            this._columnTiles = ColumnTiles;
                            this._context2D = null;
                            this._lstTile = [];


                            //three.js  properties 
                            this._graphSceneNodeParent = null;
                            this._nodeSceneName = '';
                            this._graphSceneNode = null;
                            this._geometry = null;
                            this._material = [];
                            this._mesh = null;
                    }

}

My first question is:

Is it better to use 1 THREE.PlaneGeometry by level with tile/face inside or multiple THREE.PlaneGeometry by level with THREE.PlaneGeometry/tile?
The questionbehind this one is, Do three.js render all faces of THREE.PlaneGeometry or just visible face in the viewport?

Solution 1: using 1 THREE.PlaneGeometry by level with tile/face

async function TestcreateGeometryLevelsTile(LODImage) {

                    //get informations array (compute previously in getInfoFromURL function called in buildPyramid method)       
                    var lstlevels = LODImage._lstLevelInfo;
                    var tilesize = LODImage._tileSize;  
                    var posx = 0, posy = 0, posz = 0;

                    //main node
                    LODImage._graphSceneNode = new THREE.Object3D();
                    LODImage._graphSceneNode.name = LODImage._nodeSceneName;
                    LODImage._graphSceneNodeParent.add(LODImage._graphSceneNode);
                    LODImage._graphSceneNode.position.set(posx, posy, posz);

                    //for each level
                    var col = 0;
                    for (var i = 0; i < lstlevels.length-1; ++i) {

                            var currentlevel = lstlevels[i];

                            //get the size previously computed
                            var currentWidth = currentlevel._width;
                            var currentHeight = currentlevel._height;
                            var currentRowSize = currentlevel._rowTiles;
                            var currentColumnSize = currentlevel._columnTiles;

                            // create a plane geometry for the image with a width of 10
                            // and a height that preserves the image's aspect ratio                        
                            currentlevel.setGeometry(new THREE.PlaneGeometry(currentWidth, currentHeight, currentRowSize, currentColumnSize));   
                          

                            //create a array of materials from the lstTile context
                            var lstTile = currentlevel._lstTile;
                            var lenght = currentlevel.getGeometry().faces.length / 2;
                            var indexTile = 0;
                            var col2 = 0;
                            for (var y = 0; y < lenght; ++y) {

                                    var j = 2 * y;
                                    var currentTile = lstTile[indexTile];

                                    buildTexturedPlaneGeometry(currentlevel, i, currentTile, j, y);
                                    indexTile++;
                                    col2 += 20;
                            }

                            // Combine the geometry and material into a mesh
                            var nameMaterial = "material_" + i + "_" + lenght + "_faces";
                            var multiMaterial = new THREE.MeshFaceMaterial(currentlevel.getMaterialArray());
                            multiMaterial.name = nameMaterial;
                            currentlevel.setMesh(new THREE.Mesh( currentlevel.getGeometry(), multiMaterial));

                            buildNode(LODImage._graphSceneNode,
                                    currentlevel._graphSceneNode,
                                    LODImage._nodeSceneName, i,
                                    currentlevel.getMesh(), posx, posy, posz);

                            posx += 50;
                            col += 40;
                    }
            }

function buildTexturedPlaneGeometry(currentlevel, indexLevel, currentTile, indexvertex, indexTile) {

                    var texture = new THREE.Texture(currentTile._context2D.canvas);
                    var material = new THREE.MeshBasicMaterial({ map: texture });
                    currentlevel.addMaterial(material);
                    currentlevel.getGeometry().faces[indexvertex].materialIndex = indexTile;
                    currentlevel.getGeometry().faces[indexvertex + 1].materialIndex = indexTile;


            }

ps: by the way, i still have a warning in this solution with THREE.MeshFaceMaterial, i read that this object is replace by THREE.MeshMultiMaterial but when i’m try it my quad is black!

ps2: with this solution i have only one level rendered…the second on is black… any idea?

Thx

Sorry for code presentation new try:

class LodImage {
        constructor(scene, nodeParent, nameNode) {

                           //Image properties 
                            this._url = '';
                            this._width = 0;
                            this._height = 0;
                            this._tileSize = 0;
                            this._lstLevelInfo = [];

                            //three.js  properties 
                            this._scene = scene;
                            this._graphSceneNodeParent = nodeParent;
                            this._nodeSceneName = nameNode;
                            this._graphSceneNode = null;
                    }

                    async buildPyramid(imageUrl, tileSize, nameNode) {

                            //compute all resize global images for each level
                            await buildPyramidModel(this, imageUrl, tileSize);
                           TestcreateGeometryLevelsTile(this);                            
                    }
}


class LevelsInfos {
                   constructor(levelIndex, width, height, RowTiles, ColumnTiles) {

                            this._levelIndex = levelIndex;
                            this._width = width;
                            this._height = height;
                            this._rowTiles = RowTiles;
                            this._columnTiles = ColumnTiles;
                            this._context2D = null;
                            this._lstTile = [];


                            //three.js  properties 
                            this._graphSceneNodeParent = null;
                            this._nodeSceneName = '';
                            this._graphSceneNode = null;
                            this._geometry = null;
                            this._material = [];
                            this._mesh = null;
                    }

}

And the function to build the geometry and binding the different tile textured on the face of a plan::

 async function TestcreateGeometryLevelsTile(LODImage) {


                        //get informations array (compute previously in getInfoFromURL function called in buildPyramid method)       
                        var lstlevels = LODImage._lstLevelInfo;
                        var tilesize = LODImage._tileSize;
                      

                        //get pointer this
                        var posx = 0, posy = 0, posz = 0;

                        //creation du noed principale  
                        LODImage._graphSceneNode = new THREE.Object3D();
                        LODImage._graphSceneNode.name = LODImage._nodeSceneName;
                        LODImage._graphSceneNodeParent.add(LODImage._graphSceneNode);
                        LODImage._graphSceneNode.position.set(posx, posy, posz);

                         for (var i = 0; i < lstlevels.length-1; ++i) {

                                var currentlevel = lstlevels[i];
                                //get the size previously computed
                                var currentWidth = currentlevel._width;
                                var currentHeight = currentlevel._height;
                                var currentRowSize = currentlevel._rowTiles;
                                var currentColumnSize = currentlevel._columnTiles;


                                // create a plane geometry for the image with a width of 10
                                // and a height that preserves the image's aspect ratio    
                                var geometry = new THREE.PlaneGeometry(currentWidth, currentHeight, currentRowSize, currentColumnSize);

                             
                                //create a array of materials from the lstTile context
                                var lstTile = currentlevel._lstTile;                            
                                var lenght = geometry.faces.length / 2;
                                  var indexTile = 0;
                                var col2 = 0;
                                var materials = [];
                                for (var y = 0; y < lenght; ++y) {

                                        var j = 2 * y;
                                        var currentTile = lstTile[indexTile];       

                                        var texture = new THREE.Texture(currentTile._context2D.canvas);
                                        texture.needsUpdate = true;
                                        var material = new THREE.MeshBasicMaterial({ map: texture });
                                        currentlevel.addMaterial(material);
                                         geometry.faces[j].materialIndex= y;
                                        geometry.faces[j+1].materialIndex= y;

                                        indexTile++;
                                   
                                }

                                // Combine the geometry and material into a mesh
                                var multiMaterial = new THREE.MeshFaceMaterial(currentlevel.getMaterialArray());
                                 currentlevel.setMesh(new THREE.Mesh(geometry, multiMaterial));



                                buildNode(LODImage._graphSceneNode,
                                        currentlevel._graphSceneNode,
                                        LODImage._nodeSceneName, i,
                                        currentlevel.getMesh(), posx, posy, posz);


                               posx += 50;
                        }
                }

Edit: i just noticed that i had also a updatetexture pb…Do you think i need to do a promise on each multimaterial or on each materials?
Do you think it is another pb?
So in fact, i think my ps2 is related to this pb!
Any idea?

Thank again!

ps: by the way, how do i solve a topic? i didn’t findthe button? sorry i’m new on this forum… :):sweat_smile:

Hi,

So i tried to refactorized my code, but still…not working…
in this version i’m rtying to get each part of my drawpyramid as a promise to be sure that everything is one in the good timing. Nevertheless, it seems that when i’m doing the node linking with the mesh there 'is still a pb of material update or something like this…
i got this message when it comes the moment i call the “buildNodePromise” function :

TypeError: sphere is undefined

as far as i know i could find something like this on my research
https://stackoverflow.com/questions/32039553/three-js-typeerror-cannot-read-property-center-of-undefined

but i don’t understand the sproposed solution… my geometry and my material are correctly done…I don’t understand whyyyyyyy! :sob:

please help! :slight_smile:
thx


function loadTilesIntoMaterialsPromise(lstTile, geometry, ) {

    return new Promise(async function (resolve) {

        var materials = [];
        var lenght = geometry.faces.length / 2;
        var indexTile = 0;    
   
        for (var y = 0; y < lenght; ++y) {

            var j = 2 * y;
            var currentTile = lstTile[indexTile];

            //construct a plan with different color by tile (location: ThreeEngineUtils.js)
            await buildTexturedPlaneGeometryPromise(geometry, materials,
                currentTile._context2D.canvas, j, y);

            indexTile++;
        }

        resolve(materials);
    })
        .then(function (res) { return res; });
}

drawTilesPromise will basicaly bind the textures on geometry to give us a mesh



function drawTilesPromise(geometry, materials) {
    return new Promise(async function (resolve) {

        // Combine the geometry and material into a mesh
        var multiMaterial = new THREE.MeshFaceMaterial(materials);
        var mesh = new THREE.Mesh(geometry, multiMaterial);
        resolve(mesh);
    })
        .then(function (res) { return res; });

}

The drawPlanPromise will just build a THREE.PlaneGeometry in a promise just to be sure that it is built before other functions

function drawPlanPromise(LODImage, indexLevel) {
    return new Promise(async function (resolve) {


        var currentlevel = LODImage._lstLevelInfo[indexLevel];

        //get the size previously computed
        var currentWidth = currentlevel._width;
        var currentHeight = currentlevel._height;
        var currentRowSize = currentlevel._rowTiles;
        var currentColumnSize = currentlevel._columnTiles;

        var geometry = new THREE.PlaneGeometry(currentWidth, currentHeight, currentRowSize, currentColumnSize);

        resolve(geometry);
    })
        .then(function (res) { return res; });

}

This function construct a promise to build node hierarchy between parent (which can be any of THREE.Object3D : node, scene…)

function buildNodePromise(parent, nameNode, indexChild, childmesh, posx = 0, posy = 0, posz = 0) {
    return new Promise(function (resolve) {

        var name = generateLevelName(nameNode, indexChild);
        var childNode = new THREE.Object3D();
        childNode.name = name;
        parent.add(childNode);
        childNode.add(childmesh);
        childNode.position.set(posx, posy, posz);

        resolve(childNode);
    })
        .then(function (res) { return res; });

}

this method build all needed rendering objet to display one level of my image pyramide

function drawLevelPromise(LODImage, indexLevel, posx, posy, posz) {
    return new Promise(async function (resolve) {

        //access simplifaction
        var currentlevel = LODImage._lstLevelInfo[indexLevel];
        var lstTile = currentlevel._lstTile;

        // create a plane geometry for the image     
        var geometry = await drawPlanPromise(LODImage, indexLevel);

        //load all tiles into materials bind to plan faces
        var materials = await loadTilesIntoMaterialsPromise(lstTile, geometry);

        //load all tiles into materials bind to plan faces
        var mesh = await drawTilesPromise(materials, geometry);

        // //link the mesh to a node in the scene
        var node = await buildNodePromise(LODImage._graphSceneNode,
            LODImage._nodeSceneName, indexLevel,
            mesh, posx, posy, posz);      
        currentlevel._graphSceneNode.add(node);

        resolve(true);
    })
        .then(function (res) { return res; });

}

function to create the main node of LODImage


function buildLODImageNode(LODImage, posx, posy, posz) {
    return new Promise(async function (resolve) {     
        //creation du noed principale  
        LODImage._graphSceneNode = new THREE.Object3D();
        LODImage._graphSceneNode.name = LODImage._nodeSceneName;
        LODImage._graphSceneNodeParent.add(LODImage._graphSceneNode);
        LODImage._graphSceneNode.position.set(posx, posy, posz);

        resolve(true);
    })
        .then(function (res) { return res; });
}

main function

function drawPyramidPromise(LODImage) {
    return new Promise(async function (resolve) {

        //get pointer this
        var posx = 0, posy = 0, posz = 0;

        //creation du noed principale  
        await buildLODImageNode(LODImage);

        var lstlevels = LODImage._lstLevelInfo;     
        for (var i = 0; i < lstlevels.length - 1; ++i) {

            //get the size previously computed
            await drawLevelPromise(LODImage, i, posx, posy, posz);

            posx += 50;
        }
        resolve(true);
    })
        .then(function (res) { return res; });
}

Hey @hugox,

It’s a really cool idea. Here are a few pointers:

  • not that many people know Seadragon (and OpenSeadragon), so it would be useful if you demonstrate a bit more what you are trying to do
  • many questions which you have can be answered by reading documentation. Have a look at https://threejs.org/docs/index.html#api/en/materials/Material.side for example
  • have a look at the preview section of your post, you can get a good idea what it will look like before it’s posted. This will help you format your code better.
  • when you post your code for someone else to look at - it’s usually a good idea to strip it down to bare bones, just what’s relevant to the question itself.
  • it’s easier to get people to understand your problem if you post a JSfiddle or CodePen or something similar

all the best!

hi Usnul,

Thx for your comments and tips! :slight_smile:
As i tried to explain in my first (hugly) post, I’m trying to reproduce a large image (more than 50000*50000)viewer. I’d like to be able to display and modifie with drawing tools a segmentation map on this image. So basicaly, i thought about canvas2d load into texture displayed on a plane.

So the first step is to construct a pyramidal image:
-1 construct “levels” = different downsampled image from the original one
-for each levels crop downsampled image into tiles

Then comes the rendering :
For this part I thought about two solutions.
The first one :
-Pyramid is just a node containing nodes for levels
-levels are just nodes containing a list of tiles and a THREE.PlaneGeometry with row and column defining number of tile inside the plane.
-tiles are just materials containing texture from 2d canvas of the tile model built during the pyramid construction

  • : drawing is easier you draw on only one plane
  • : you don’t have to worry about the tiles positions
  • : i think you loose the power of z-buffer algo with the graph scene because it will render all the visible plan and all geometry faces even the hidden ones. And i want to display only visible tiles.

The second one:
-Pyramid is just a node containing nodes for levels
-levels are just nodes containing a list of tiles.
-tiles are node with THREE.PlaneGeometry and a material containing texture from 2d canvas of the tile model built during the pyramid construction

+: i think that, with this solution all hidden tiles are not rendered du to graphScene
-: you have to take care of each tile position
-: drawing is harder you have many tiles and levels

for both solutions, i just discovered that you have some limitations on live canvas numbers. So I don’t know for now on, if i can do it with one canvas/tile…
Moreover, I think that the loading and saving of all tiles into canvas is not memory efficient. I think using blob could be better but my knowledge of this part is poor for now. So i’m focusing on the first points and then i will try blob stuff! :slight_smile:

I hope this explaination is better than the previous one!
I will try to set a fiddle soon but i’m still facing many synchronisation pb to draw my tiles and i want to finish both solutions.

Anyways, all comments are welcome! :slight_smile:

hey @hugox,

Okay, I understand your problem. I suggest you read up on 2 topics:

  • QuadTree. Good starting point is wiki
  • (Sparse) Virtual Texture (aka MegaTexture). There are a few good resources out there, here’s wiki, there’s also a bit-rotten three.js implementation on github

Your key problem is the bottom of the pyramid. You need to make sure to load relevant tiles of the bottom portion of the pyramid depending on the zoom level. QuadTree is going to be very useful for that, as it offers a very natural mapping between a quad and an image tile. VirtualTexture is an overkill for just viewing images on a plane, as it’s a general-purpose texturing solution, but you might find it more interesting to do :slight_smile:

That’s not very important here. Your problem is not at the top of the pyramid, but at the bottom. Textures 16k+ aren’t really supported by GPUs, you’re on your own there, you need some kind of software solution here.

That’s not really the case. Canvas is less efficient than webgl API, but memory usage will be similar. Blob might be ok, but using Uint8Array is probably most efficient (you can do even more with compressed textures, but I wouldn’t recommend it as a starting point).

hi,

I don’t think Quad tree will be usefull for this kind of application.
Basicaly, the building part of pyramid could be done in different ways but the simpliest one i think is Zoomify algo, which is the one i explained in my previous post.
So in fact, you have already all your tiles and levels as datas.
You just need to render them.
Virtual texture is lesser good than openSeaDragon http://openseadragon.github.io/ and use only canvas and blob adress to render visible tiles.
I opened 125000*125000 image with openseaDragon without any latency! :slight_smile:

that’s why i talk about second solution with one plane by tile and a tilesize squared equivalent to 128 squared.

as far as know, openseaDragon doesn’t seems to have memory pb or limitation because it is based on the use of blob link to a canvas. so they load only what they need to load at the wanted timing. my idea was to bypassing the canvas with textures. buti noticed on another project the easiness to draw directly on texture with canvas that’s why i kept canvas.
And I said, i 'm not very confident with blob adress linked to texture in js for now. :slight_smile:
Plus, i don’t know if it’s possible to draw on texture with three.js functions without using canvas obviously?

to sumerise, i want to draw a tilemap of small tiles all with textures! :wink:

After a bit of digging, original zoomify paper uses quad-tree, just doesn’t call it that.
image

In the paper they don’t state that they link tiles in explicit hierarchies. Modern Virtual Texture implementations seem to all explicitly use QuadTree to keep track of the hierachy and speed up access, some incorporate memory management directly into the tree also (loading/unloading tile data).

yes i already read it,

it something very similar to quadtree mecanism but it s just the model part not the rendering one!
All the tiles are precomputed and store with namespace and the rendering function will just take the adress and get the correct tiles at the correct level depending on zoom level and viewport colision with the tiles nodes. I didn’t try modern virtual texture for now but it seems very promising!

Edit : by the way do you know how to mipmap from adress pointer considering that all the mipmap are done in previsous step?
And do you know if it s possible to clipmap in the same way?

thx you