How to use the `Node` classes instead of TSL?

[trusktr]

For example, can we use the form new RotateNode(position, rotation) instead of rotate(position, rotation)? I’m curious what that looks like. It doesn’t seem to be documented. Examples all use TSL.

The class form seems more flexible, for example easier to dynamically modify the connections between nodes of a node graph:

const rotateNode = new RotateNode()
rotateNode.positionNode = positionNode
rotateNode.rotationNode = rotationNode

// later, re-wire the the position node:
rotateNode.positionNode = newPosNode

It seems like this format is perhaps better for dynamic scenarios, such as a node graph UI modifying nodes on the fly (drag/drop connection from one node to another).

But the rotate() form does some special magic with ShaderNodeProxy, so I’m not sure what’s missing with the new RotateNode() form.

[trusktr]

first attempt (no success yet)

For starters I tried to port this TSL example,

const map = new THREE.TextureLoader().load('...')
map.offset.x = -0.5

const uMapMatrix = uniform(map.matrix)

var opacityNode = Fn(() => {
	const vMapUv = uMapMatrix.mul(vec3(uv(), 1))
	const edgePos = vMapUv.x
	const pixelWidth = fwidth(edgePos)
	const fade = pixelWidth.mul(1.5)
	const alphaFactor = smoothstep(fade.negate(), 0, edgePos)
	const newOpacity = materialOpacity.mul(alphaFactor)
	If(newOpacity.lessThanEqual(0), () => Discard())
	return newOpacity
})();

var cubix = new THREE.Mesh( new THREE.BoxGeometry(1,1,1),
  new THREE.MeshPhysicalNodeMaterial( { opacityNode } ) );

(example on codepen using TSL only)

to use only Node classes directly, not mutating any nodes only using their constructors to start with, but was unable to full convert:

const map = new THREE.TextureLoader().load('...')
map.offset.x = -0.5

// Type is not automatically inferred with manual nodes like
// UniformNode (maybe it should be?)
const uMapMatrix = new UniformNode(map.matrix, 'mat3')

// Unable to convert Fn() to manual FnNode because FnNode is
// not exported from 'three/webgpu' (it is not even exported
// from the original src/nodes/tsl/TSLCore.js file). Is this
// an oversight in src/nodes/Nodes.js? Or is it intentional?
var opacityNode = Fn(() => {
	// This is not quite the same as Const() in TSL,
	// and the type is not automatically inferred
	const one = new ConstNode(1, 'uint')

	const uv = new AttributeNode('uv', 'vec2')

	// There is no equivalent Vec3Node (maybe there should be), instead the
	// vec3() TSL function does a dance with ConvertType to create the sub-node
	// graph with inferred types and a JoinNode. Perhaps a Vec3Node class should
	// do a similar dance to enable the equivalent Node class, for consistency.
	const uvVec3 = new JoinNode([uv, one], 'vec3')
	
	// Unable to convert mul() to OperatorNode because OperatorNode
	// is not exported from 'three/webgpu' (but is exported as default
	// from src/nodes/math/OperatorNode.js). Is this an oversight? Or intentional?
	const vMapUv = uMapMatrix.mul(uvVec3)
	// const vMapUv = new OperatorNode('*', uMapMatrix, uvVec3)
	
	const edgePos = vMapUv.x
	
	// Unable to convert fwidth() to MathNode, MathNode is not
	// exported from three/webgpu (but is exported as default from
	// src/nodes/math/MathNode.js). Is this an oversight?
	const pixelWidth = fwidth(edgePos)
	
	// Unable to convert mul() to OperatorNode because OperatorNode
	// is not exported from 'three/webgpu' (but is exported as default
	// from src/nodes/math/OperatorNode.js). Is this an oversight? Or intentional?
	const fade = pixelWidth.mul(1.5)
	// const fade = new OperatorNode('*', pixelWidth, new ConstNode(1.5, 'float'))
	
	// Unable to convert negate() to MathNode, MathNode is not
	// exported from three/webgpu (but is exported as default from
	// src/nodes/math/MathNode.js). Is this an oversight?
	const negFade = fade.negate()
	
	const zero = new ConstNode(0, 'uint')
	
	// Unable to convert smoothstep() to MathNode, MathNode is not
	// exported from three/webgpu (but is exported as default from
	// src/nodes/math/MathNode.js). Is this an oversight?
	const alphaFactor = smoothstep(negFade, zero, edgePos)
	
	// Unable to convert mul() to OperatorNode because OperatorNode
	// is not exported from 'three/webgpu' (but is exported as default
	// from src/nodes/math/OperatorNode.js). Is this an oversight? Or intentional?
	const newOpacity = materialOpacity.mul(alphaFactor)
	
	// Not even sure how to convert If() to a Node class, and how
	// to manage the special stack (why is that required?).
	If(newOpacity.lessThanEqual(0), () => Discard())
	
	return newOpacity
})();

var cubix = new THREE.Mesh( new THREE.BoxGeometry(1,1,1),
  new THREE.MeshPhysicalNodeMaterial( { opacityNode } ) );

(example on codepen trying to use Node classes directly)

The main issues I see are

• some nodes are not exported at all even from original source files, that makes using plain Node classes impossible
• some node classes are exported fro original source files, but not from the index (three/webgpu).
  • Importing these from src/... will cause a fork in the library that will import multiple versions of Three.js APIs
• certain features like automatic type inference are a feature of TSL only, not the node classes.
  • If the Node classes had the inference instead, the APIs would be more consistent with each other, and TSL would be a thinner wrapper
• I don’t know what certain things like nodeProxy, nodeProxyIntent, nodeObjectIntent, etc, are doing, but I was able to omit them (f.e. using JoinNode directly for uvVec3 without the nodeObjectIntent call that vec3() calls)
  • I don’t know if this causes some issue, but it also seems like a TSL-only feature that is also not consistent with Node classes

Overall, I think if

• logic from TSL is moved into node classes,
• and all Node classes are exported and easy to use (f.e. auto type inference),
• then TSL becomes a thinner wrapper over the classes,
• and most importantly, making a mutable node graph becomes easy

Benefits of “a mutable node graph”

The idea is that the node graph of a material could be modified after it has been created, for example by disconnecting nodes, reconnecting them, adding nodes between nodes, etc.

Right now, the graph is created once only (typically with static TSL code), then set in stone (via setup()), and none of the examples deal with the underlying graph directly. Despite the term “nodes” being prominent in the contept of TSL, the overall node graph is more of an internal result at the moment.

There could be something akin to the old onBeforeCompile, for example, but that receives the material’s node graph, where we would be able to traverse the graph, modify/replace nodes, inject nodes, etc.

mapping mutable nodes to mutable UI nodes

It isn’t clear how the currently somewhat-static node graph (based on the limitations I commented in the conversion example), can cleanly map to a mutable GUI node graph in a one-to-one fashion.

Maybe it is for this reason that the node playground is currently missing certain nodes. For example, I am not able to find If or Fn in the list of nodes, so the visual paradigm differs a bit from the TSL paradigm.

What I’m imagining is that, if we were able to control the graph purely with mutable Node instances (currently some Node classes are not even exported), then

• a thinner lighter TSL would be implemented on top without certain features being TSL-specific, those features would be in the Node classes directly
• a mutable node graph would more simply map to a mutable GUI graph, there’d be nothing that would prevent any of the current features from appearing as a node in the node list (f.e. Fn, If, Discard, etc).
• it would become easier to take any node graph and serialize, deserialize, modify, repeat.
• a three-specific GUI would become a thin wrapper, without necessarily even having to be mapped to from non-GUI nodes. Nodes themselves could contain all of the information for the GUI including all input/output connections (the graph itself).

I would like to see the full expressivity of a mutable node graph for those reasons.

Has going in this direction already been discussed anywhere?

[dubois]

@trusktr

There could be something akin to the old onBeforeCompile, for example, but that receives the material’s node graph, where we would be able to traverse the graph, modify/replace nodes, inject nodes, etc.

Why can’t you just do your modifications and call setup again?

I don’t quite understand what you are trying to do. Also, I haven’t actually used TSL, but it seems like you are describing the internals of this system, which are not meant to be used by the consumer.

Essentially, you are describing a DAG. You can describe this DAG with json, you don’t need any internals of this library to do so. It’s a type of node (like add, multiply, step, whatever) and it has parents and children conceptually. Some inputs (like value and edge in step) and some outputs (like, the result of add, or perhaps multiple outputs if it’s a function). All of these are just references to other nodes.

So essentially, if TSL can be serialized, you could theoretically write it in that format. That way you would not even care about the public facing classes even, as long as there is a type “add” and it takes two operands you could write a json that could theoretically be compatible even if TSL internals change.

[trusktr]

Is there a good example you can link?

[dubois]

I’m just thinking about this conceptually. You never needed onbeforecompile if you just wrote used one of the templates in a shader material. Basically you would compile it then and there, when you make the material.

Thus, why would you need a hook like that here? Just change the graph, make a new material, go on with life

[trusktr]

I’m not sure what you’r describing to do exactly. Are you suggesting to copy/paste the code out from inside Three material, so I can then modify it? If so, then yeah that would work, but I’d like to avoid that for various reasons.

I believe that, at least in theory, one of the points of TSL and a “node graph” is to be able to modify that node graph.

Of course I can recreate a whole node graph, but I’d like to see the alternatives to that.

Also I’d like to see what is possible without TSL (just pure nodes), because one of my goals is to make a node GUI, and the TSL-specific function closures seem to be something not representable purely with a node graph (or is it?).

[trusktr]

@dubois

So essentially, if TSL can be serialized, you could theoretically write it in that format

That’s an interesting idea, although the downside of that is having to write/modify in a format that isn’t the standard developer interface, but moreso a serialization layer for import/export.

I think it may be a good idea to look at that JSON and see what final node graph some TSL outputs, and maybe that’ll give me insight into how to write up Nodes programmatically.

I’m just thinking that it’d be great to be able to wire up nodes fairly easily without TSL (ofc not as concise as with TSL), and that this gives the benefit to making node GUIs.

[trusktr]

@dubois Doh! I updated the OP, I had forgotten to link to the codepen example that tries to use Node classes only. I commented which parts I was not able to convert from TSL.