Recently I’ve been following the new WICG html-in-canvas discussions and experiments, and honestly this feels like a pretty major shift for using it instead of html2canvas, Could this become a future replacement for html2canvas-based texture systems like Render live Html directly into textures and UI panels inside the 3D scene.
What are your thoughts about this direction?
Also,
why does almost nobody here seem to be talking about this yet?
it’s very new, I’ve been following the progress too and it looks promising
Exactly, i remember not so long ago i needed and still need actually implementation of heavy and many instance of Live canvas HTML content, i have to try it
threejs actually has its own implementation using the HTML-in-canvas api…
but as it’s still an experimental feature you need to enable CanvasDrawElement in chrome’s experimental flags, there doesn’t seem to be any existing browser support table for this, the mozilla team also don’t seem keen on adopting the specification…
Is this related to the “svg hack” where you can embed HTML in an svg and render it to canvas?
const canvas = document.getElementById('myCanvas');
const ctx = canvas.getContext('2d');
// 1. The HTML you want to render
const data = `
<svg xmlns="http://www.w3.org/2000/svg" width="200" height="200">
<foreignObject width="100%" height="100%">
<div xmlns="http://www.w3.org/1999/xhtml" style="font-size:40px">
<em>Hello</em> <span style="color:red">World</span>
</div>
</foreignObject>
</svg>`;
// 2. Turn it into a URL
const svg = new Blob([data], {type: 'image/svg+xml;charset=utf-8'});
const url = URL.createObjectURL(svg);
// 3. Load into an image and draw
const img = new Image();
img.onload = () => {
ctx.drawImage(img, 0, 0);
URL.revokeObjectURL(url);
};
img.src = url;
oh cool, I didn’t know about that hack, does it support HTML interactions? in essence I think not though it’s using the HTML-in-canvas API which is an experimental chrome feature that doob has integrated into threejs here… Added HTMLTexture by mrdoob · Pull Request #31233 · mrdoob/three.js · GitHub
Wow Mozilla’s pushback is wild “you can put html in front of a canvas”
and to use multiple layered canvases 
Their reasoning for it is somewhat valid as it exposes what the user is being shown on the DOM and potentially becomes a way to easily hijack inputs with XXS.
HTML-in-canvas is violating the principle that JS should not see what is being shown to the user.
as well as there being direct issues with cross site origin rendering using iframes…
I think for the most part though they may be overwhelmed by the work it’d take to align a thorough and correct specification to make this fully safe and compatible, it’s quite a big task and is a huge change, it’s a bit of a “can’t be bothered” retortion.
there was also a mention of “pixel manipulation” whereby rendered text could be manipulated in the canvas to say or mean something else to what was originally written in plain HTML.
they definitely have a few valid points but I agree there are also quite blunt and silly elements to their push back
I didn’t think about manipulating text, that actually sounds pretty legitimate. I think making a UI library using slug is the way to go
definitely a valid option if there was a communal push to encompass events within that, in a cross compatible library, I think HTML-in-canvas is also viable but will take a lot of work and will likely need limiting to certain types of non sensitive dom elements, that being said, there’s a lot of variability to how unaware developers could include elements that could be a cause for concern…
There’s definitely a void that needs bridging to include styling, layouts, fonts, kerning, line / word / letter wrapping, (even forms and buttons), events, etc directly in canvas with more support than what currently exists, it may take a while but I do think the full potentials of a system like this will be unlocked from a lower level browser specification than a timultuous sand castle on top of existing spec being built, maybe?
Actually I’m a bit confused with text changing concern. I feel like I might be imagining something much less than what they are.
So there’s a button in my HTML that says “click here to give me your social security number and all your credit cards”.
Then I put it in WebGL and say “click here to shoot flower heart teddy bears”.
Like, why can’t I just put the second in my by html to begin with, but make it like… do the first?
Am I understanding this right?
from what i understand it is not just about text changing it is about data hijack Like @Lawrence3DPK said about the XXS attacks, XSS is js attack and html-in-canvas allow data to be collected from the canvas witch mean JS sees things and content it suppose to not sees it by getImageData for example … that is why svg uses foreignObject to block it.
But like, does html-in-canvas allow for that, or oyu can grab the data today?
good question, I’m not at all familiar with the methods used in XXS (cross site scripting), man in the middle attacks (man in the browser) or DOM spoofing but my understanding is yes the canvas is currently vulnerable and exposed to being intercepted before it reaches the legitimate user and changed before rendering…
it’s essentially monkey patching eg…
const originalFillRect = CanvasRenderingContext2D.prototype.fillRect;
CanvasRenderingContext2D.prototype.fillRect = function(x, y, w, h) {
// modify behavior
console.log("Intercepted draw");
// call original
return originalFillRect.call(this, x, y, w, h);
};
or…
const originalGetContext = HTMLCanvasElement.prototype.getContext;
HTMLCanvasElement.prototype.getContext = function(type, ...args) {
const ctx = originalGetContext.call(this, type, ...args);
// wrap or modify ctx
return ctx;
};
and post render pixel manipulation…
const imageData = ctx.getImageData(0, 0, width, height);
// modify pixels
imageData.data[i] = 255;
ctx.putImageData(imageData, 0, 0);
you can if you’re OK with lawsuits, I think the problem is that if someone else intercepted the message before being shown to the user you could potentially end up in a lawsuit that wasn’t the fault of yourself…
It’s definitely interesting to follow the progress of this specification in that regard as it could change the way things are rendered entirely if a successful proposal is put together, we could be looking at something other than html5 
This is an interesting topic, if you can elaborate more I think it would be great. My understanding was that this just allows you to use blink for example, and if it is rasterizing stuff, maybe those buffers, but I don’t understand why certain sensitive fields would be included, or maybe I’m thinking that a form wouldn’t even be there. This is more for information - text, and sliders and buttons I think.
I don’t really know enough to elaborate but it seems these issues are what the mozilla team are concerned about.
here are some further resources on the subject of intercepting and manipulating canvas data…
there is also a “Privacy-preserving painting” section in the github repo that has some interesting information…
I don’t know if this means these elements can be added and therefore should not be or that the api dissalows adding such sensitive fields and elements…
I use my “html_to_canvas” (github: polygontwist) implementation for text, image, and table output, as well as interactions with buttons, radio buttons, and checkboxes. I can simply use HTML and CSS without having to redefine everything from scratch.
Three years ago, I hadn’t found anything like this yet—it’s great to see that others are exploring this as well.
Hi @manthrax …
I sent to You and You answered:
Niice. :). Thanks for sharing it! glad you found that code useful.
So I decided to post here:
Anyone still here.. Google IO Dev Keynotes showed this:
Interesting that they doubled down on the text selection and copying.
