3d-Tiles-Renderer: A 3D Tiles implementation for three.js from NASA JPL

Hey guys! It’s been around for a while now but I wanted to share the 3d-tiles-renderer project I worked on and open sourced years ago from NASA JPL! For a couple of our Mars missions now we’ve been producing 3d tiles data sets that include data from our orbiters and rovers which are used in some of our tools for displaying contextual landscape and environment information for rover operations. The 3d tiles renderer here has specifically been used on the Perseverance and Curiosity rover missions over the last few years!

For those who don’t know 3d tiles is a format designed to support progressive and scalable loading of large datasets like cities or topological maps (of Mars data!) so only a minimal amount of data is displayed within some error metrics. This is done by producing various levels of tile LoDs in a bounding volume hierarchy until the full level of detail is represented at the leaves.

There are a number of demos at the repo to check out include one that renders a Dingo Gap data set from Curiosity. The project is also used in the Explore with Perseverance Experience to display the terrain from the mission!

Explore with Perseverance Website

Dingo Gap Data Set

22 Likes

Hey @gkjohnson,

Google Maps recently added the Map Tiles API for retrieving photogrammetry tiles of Google Maps in their API toolset. I tried rendering some of these tiles via your 3DTilesRendererJS with no luck yet, as Google requires an API key for every request to each tile. Do you think you’ll add support for this new update to the Google Maps API? I can definitely look into a way it could work when using your Tiles Renderer. I think using this endpoint within a three environment would be a really useful and powerful addition to any workflow utilizing these resources. Let me know your thoughts.

That would be awesome - it would be great to see a demo with the Google Earth data.

as Google requires an API key for every request to each tile. Do you think you’ll add support for this new update to the Google Maps API?

It should hopefully already be possible. Cesium Ion is supported and requires API keys and other added header values / query parameters, as well. You can see the TilesRenderer.fetchOptions field and TilesRenderer.onPreprocessURL which allow you to change the fetch options and url for each piece of data loaded.

Here’s roughly how it’s used for Cesium ion data:

const version = url.searchParams.get( 'v' );
tiles = new TilesRenderer( url );
tiles.fetchOptions.headers = {};
tiles.fetchOptions.headers.Authorization = `Bearer ${json.accessToken}`;

tiles.onPreprocessURL = uri => {

  uri = new URL( uri );
  uri.searchParams.append( 'v', version );
  return uri;

};

If there are still problems feel free to make an issue and we can work through it. And please do share your demo! :grin:

2 Likes

I managed to get the gltf files to download in this PR: WIP: Aiden/gmaps by aiden-jeffrey · Pull Request #330 · NASA-AMMOS/3DTilesRendererJS · GitHub. However there’s nothing visible in the scene. @gkjohnson - are you able to give me some general pointers?

2 Likes

If anyone is still interested in helping out with any of the 3d tiles globe work I made an issue here to document some of the things that I’d like to get done in the example in case anyone wants to pick something off. Aiden got a good demo started but it still needs some work to provide a more google-maps-like globe experience with panning and zoom in. I plan to put a little more time into it when I get the chance:

2 Likes

I’ve released v0.3.19 of the 3d-tiles-renderer the other day which adds some fixes and additional support for Google’s 3d tiles including a convenience class to easily load and add a globe into your scene. The next release will include some fairly significant optimizations that improve download and memory performance with the globe tile set quite a bit.

Globe-style controls are the big thing that’s still needed. I don’t know if I’ll be able to work on those myself but if anyone would like to look into provide a Google Earth or other type of globe control system it would be much appreciated!

Tokyo Tower

Sample Initialization

5 Likes

Just released v0.3.20 - this version brings some bug fixes in addition to some frustum culling optimizations that benefit Google Tiles pretty heavily. In some cases over 45% fewer tiles are drawn per frame and over 35% fewer tiles are downloaded which can significantly improve both the CPU and GPU performance of the renderer!

v0.3.20 Release Notes

The Colosseum

5 Likes

I’ve added a couple more examples that make tile loading more smooth and will make navigating the Photorealistic Tiles globe a lot more intuitive! :earth_americas:

The code for both examples is currently in the example/src directory. They may be merged into core in the future.

GlobeControls

Showing globe navigation

GlobeControls provide a Google Earth-like controls system which makes navigating the globe a lot more simple (and possible). Looking forward to seeing what people make with this! You’ll need your own API key but you can see the demo here.

Fade Transition Tiles

Showing tiles loading with and without fading to demonstrate the difference

This demo uses a complementary dither to fade tiles in and out, preventing some of the “popping” effect that occurs when new tiles load in or the camera moves. It’s based on some of the ideas from this Cesium article with some additional improvements including fading tiles out more quickly when the camera moves to prevent the amount of fading tiles to acummulate too much.

You can see the demo here

3 Likes

Version 0.3.36 of 3d-tiles-renderer has been released which brings support for a new plugin architecture and plugins, support for some new extensions, orthographic camera support for GlobeControls, and more! This release also brings quite a few new improvements thanks to the Cesium GIS Ecosystem Grant which aims to support improving projects in the geospatial visualization community! You can check out my post on the improvements here.

See the full v0.3.36 changelog here.

Orthographic GlobeControls

This wound up being significantly more complicated than I originally thought but I think it feels pretty nice, now. The GlobeControls class now supports orthographic as well as perspective cameras so you can change views based on what’s needed. Demo here.

Orthographic Camera Transition

The example camera transition manager synchronizes the perspective and orthographic camera position and animates the transition between the two camera views when toggling. Demo here.

Alternate Tile Set Textures

A example and plug have been added to demonstrate how to load a separate set of textures for an existing tile set. This can be used for loading and displaying pre-generated data overlays like this slope magnitude data set for generated Martian imagery. Demo here.

GLTF Metadata Extensions

The glTF EXT_structural_metadata and EXT_mesh_features allow for storing and querying feature information from geometry attributes and textures as well structured data definitions so data such as building properties and materials can be displayed. Demo here.

More Plugins

A number of extension classes have been migrated to plugins including Cesium Ion and Google Cloud authentication, tile fading, tile geometry compression, and more. See the PLUGINS docs page for more info.

10 Likes

The next version of 3d-tiles-renderer has been released! It includes quite a few changes and improvements outlined here but there most interesting are probably the ability to cap the tile geometry cache based on bytes used and inertial movement being added to GlobeControls! The inertial movement lets the globe continue spinning and slow to a stop once the user has stopped dragging, which I think adds quite a bit to the user experience.

Globe Controls Inertial Movement

10 Likes

Version v0.3.39 of 3d-tiles-renderer has been released! The big new addition is official support for @react-three/fiber so you can easily bring 3d tile sets and controls into any react scene and take advantage of the ecosystem like @react-three/drei and the postprocessing stack.

You can check out the set of examples here:

Full changeset is here, and documentation for the r3f components is here!

react-three/fiber 3D Tiles

A 3d tiles scene with several drei components and materials

A simplified look at the JSX used for a basic scene

react-three/fiber Google Photorealistic Tiles

Google Photorealistic Tiles in @react-three/fiber with an arrow pointing to NASA JPL

Simplified sample code for rendering a globe
4 Likes

This is something I’ve been looking for for a long time. So great. Very grateful for the work you have done :star_struck::star_struck:

1 Like

This is very similar to my Mars app. I haven’t processed the earth maps yet. Since these are available in a significantly higher resolution, the result would be better than with Mars.
@gkjohnson do you know if there are better maps of the entire Mars somewhere than the one I’m currently using? I use 131k but at least 4x would be nicer. I would even be very happy about double the resolution

1 Like

All Mars Resources - NASA Science :eyes:

There are hq texture maps in those resources, here’s an example of them in use… Mars Trek

And a version by “cosmic cartographers” MurrayLab which allegedly…

is a 5.7 trillion pixel image of the surface of Mars, rendered at 5.0 m/px.

https://murray-lab.caltech.edu/CTX/V01/SceneView/MurrayLabCTXmosaic.html

3 Likes

@L_ng_Tu_n_Anh

This is something I’ve been looking for for a long time.

Glad to hear it! Please post your work if you make something :grin:

@Attila_Schroeder

do you know if there are better maps of the entire Mars somewhere than the one I’m currently using

I don’t what data set you’re referring to but there’s not a straight forward answer to what the “best” map of Mars would be. There are a lot of cameras that have been used to capture surface data that will have different resolutions, some with color and some black and white, some data sets with different coverage, etc. Any equirectangular-style (assuming that’s what you’re using) or more complete view of the surface will be from stitched together from versions of these images sometimes from different data sources, have false color applied, resolution normalized and whatnot in order to achieve some goal.

You can see in the Mars Trek link from Lawrence that B&W images are tiled in and include stitching artifacts with different lighting (probably different times of day / year) likely to prioritize the highest resolution of real data over being processed to look nice:

The 3D Mars data sets shown in this project are some that my lab had developed when I still worked at JPL and merged both satellite imagery & local rover imagery (images taken from the ground perspective by our spacecraft) to generate sections of terrain for mission ops. If I recall correctly a lot of that imagery from the Curiosity rover is black and white so those images would have some false color applied when merging the data. On Mars 2020 most if not all of the cameras are in color. Unfortunately, for a variety of reasons, it’s fairly difficult to align all the rover images with the satellite imagery so I don’t think there’s a complete data set with all the available highest resolution imagery.

Some of the resources Lawrence posted look good but PDS is where all of the raw and (somewhat) processed images from the rover are. And Arizona University catalogs the imagery from the HiRise cameras here. I’m otherwise not sure what the publicly available options are for ready-to-use complete images of the full surface. And unfortunately I’m not familiar with all the Mars missions and cameras so I can’t tell you what missions or data sets to look at if you’re more interested in “real color” imagery

That’s probably more than you asked for but hopefully still insightful. It’s been a number of years since I worked on that stuff so hopefully I got most of it right. When I joined JPL 12 years ago there were a few projects using similar types of using tiled geometry for the surface of Mars, then a few years after we started using 3D Tiles for the data we use in day to day science operations to lean in on open standards.

The Eyes on the Solar System project may also be interesting to take a look at, though it doesn’t look like they’re using the highest resolution of available data.

3 Likes

I’m looking for the map material for this, my Mars.

As far as I remember, that didn’t work for you. Because of the work on the PR for indirect buffers, I haven’t gotten around to investigating the reason why it doesn’t work on Macs. Since I’m just a user and not a core developer, I need a little time to familiarize myself with the node system code. Now that the PR is finished, I’m busy with the next PR.

Lawrence3DPK’s information about Caltech was helpful. I found the downloadlink to the fileserver, 5.6 TB downloadsize. My server isn’t nearly enough for the uncompressed maps, especially if I want to turn it into a virtual texture. But I don’t care much about that at the moment. I’m just really happy that there is much better material out there than what I currently have.

1 Like