Help reproducing Unity project in Three.js

Hi there,

I’m hoping to reproduce some of the Pepper’s Cone’s project in Three.js. Essentially, they implemented a way of distorting a 3d model so that it appears undistorted when reflected from a screen onto a cone, creating a holographic effect. Their project was done in Unity and uses shaders.

I have some experience using Three.js, but mostly used their helper methods and still have some trouble with the “guts” of WEBGL, and I’m very new to shaders.

I grabbed the generated GL shader code from Unity (compiling for OPENGL ES 2.0, since I believe that’s best for WebGL) and tried to essentially rewrite their other code in JavaScript/THREE.

I shouldn’t be surprised that nothing appears on screen given my approach, but I thought I’d reach out here before… whatever my next step would be otherwise.

I used the project set-up from, and the index.js below represents the modified file in that project.


global.THREE = require('three');
const path = require('path');
const fs = require('fs');
const createApp = require('./createApp');
const createBunnyGeometry = require('./createBunnyGeometry');
const createLoop = require('raf-loop');
const getPixels = require("get-pixels")

//warp vars
var mapDiv = 4095; //0x3FFF;
var flipTexture = true;
var power = 1, alpha = 1;
var tabletScreenScale = new THREE.Vector3 (4, 3,  1);
var encodedMap;
var mapColor = [];
var outTexture;

// Create our basic ThreeJS application
const {
} = createApp();

// Create our vertex/fragment shaders
var material = new THREE.RawShaderMaterial({
  vertexShader: fs.readFileSync(path.join(__dirname, 'shader2.vert'), 'utf8'),
  fragmentShader: fs.readFileSync(path.join(__dirname, 'shader2.frag'), 'utf8'),
  uniforms: {
    time: { type: 'f', value: 0 },
    _power: { type: 'f', value: 0},
    _alpha: { type: 'f', value: 0},
    _TexRotationVec: { type: 'vec4', value: new THREE.Vector4()}

//get colours of each pixel
getPixels("", function(err, pixels) {
  if(err) {
    console.log("Bad image path")

//format pixel data into a new texture
//this function and LateUpdated are adapted from
function processPixelData(pix){

  var convertedPixels = [];
  var index = 0;

  //we could probably combine this and next loop
  for (var i = 0; i < (; i+=4){
    convertedPixels[index] = {
      g:[i + 1],
      b:[i + 2],
      a:[i + 3]

  for (var i = 0; i < convertedPixels.length; i++){
    var ec = convertedPixels[i];
    mapColor[i] = new THREE.Color("rgb(0, 0, 0)");
    mapColor[i].r = ((ec.r << LOAD_TEX_COLOR_BIT_DEPTH) + ec.g) / mapDiv;
    mapColor[i].g = ((ec.b << LOAD_TEX_COLOR_BIT_DEPTH) + ec.a) / mapDiv; 
    if (flipTexture){
      mapColor [i].g = 1 - mapColor [i].g;

  outTexture = new THREE.DataTexture( mapColor, pix.shape[0], pix.shape[1], THREE.RGBAFormat );

  material.texture = outTexture;


function LateUpdate(){
  var rot = new THREE.Quaternion();
  // although the texture's rotating eulerZ degree, the uv needs to rotate -eulerZ
  var m = new THREE.Matrix4();
  var firstVar = new THREE.Matrix4();
  firstVar.makeScale(1/tabletScreenScale.x, 1/tabletScreenScale.y, 1.0);
  var secondVar = new THREE.Matrix4();
  secondVar.compose(new THREE.Vector3, rot, tabletScreenScale);
  m = firstVar * secondVar;
  material.uniforms._TexRotationVec.value = new THREE.Vector4(m[0,0], m[0,1], m[1,0], m[1,1]);
  material.uniforms._power = 1.0;
  material.uniforms._alpha = 1.0;

// Get a nicely prepared geometry
const geometry = createBunnyGeometry({ flat: true });

// Setup our mesh
const mesh = new THREE.Mesh(geometry, material);

// Time since beginning
let time = 0;

// Start our render loop
createLoop((dt) => {


  // render
  renderer.render(scene, camera);


uniform highp vec4 _TexRotationVec;
uniform sampler2D RenderedTex;
uniform sampler2D MapTex;
uniform highp float _power;
uniform highp float _alpha;
varying highp vec2 xlv_TEXCOORD0;
void main ()
  highp vec4 tmpvar_1;
  lowp vec4 col_2;
  highp vec4 map_3;
  highp mat2 tmpvar_4;
  tmpvar_4[0].x = _TexRotationVec.x;
  tmpvar_4[0].y = _TexRotationVec.z;
  tmpvar_4[1].x = _TexRotationVec.y;
  tmpvar_4[1].y = _TexRotationVec.w;
  highp vec2 tmpvar_5;
  tmpvar_5 = ((tmpvar_4 * (xlv_TEXCOORD0 - vec2(0.5, 0.5))) + vec2(0.5, 0.5));
  bool tmpvar_6;
  tmpvar_6 = (((
    (0.001 <= tmpvar_5.x)
    (tmpvar_5.x <= 0.999)
  ) && (0.001 <= tmpvar_5.y)) && (tmpvar_5.y <= 0.999));
  if (!(tmpvar_6)) {
    tmpvar_1 = vec4(0.0, 0.0, 0.0, 0.0);
  } else {
    lowp vec4 tmpvar_7;
    tmpvar_7 = texture2D (MapTex, tmpvar_5);
    map_3 = tmpvar_7;
    bool tmpvar_8;
    tmpvar_8 = (((
      (0.001 <= map_3.x)
      (map_3.x <= 0.999)
    ) && (0.001 <= map_3.y)) && (map_3.y <= 0.999));
    if (!(tmpvar_8)) {
      tmpvar_1 = vec4(0.0, 0.0, 0.0, 0.0);
    } else {
      lowp vec4 tmpvar_9;
      tmpvar_9 = texture2D (RenderedTex, map_3.xy);
      highp vec4 tmpvar_10;
      tmpvar_10 = (_alpha * pow (tmpvar_9, vec4(_power)));
      col_2 = tmpvar_10;
      tmpvar_1 = col_2;
  gl_FragData[0] = tmpvar_1;


attribute vec4 _glesVertex;
attribute vec4 _glesMultiTexCoord0;
uniform highp mat4 unity_ObjectToWorld;
uniform highp mat4 unity_MatrixVP;
varying highp vec2 xlv_TEXCOORD0;
void main ()
  highp vec4 tmpvar_1;
  tmpvar_1.w = 1.0; =;
  xlv_TEXCOORD0 = _glesMultiTexCoord0.xy;
  gl_Position = (unity_MatrixVP * (unity_ObjectToWorld * tmpvar_1));

I’m sorry for using the forum like this–generally I try to be more informed about my problems, but I’ve been at a bit of a loss since, without help, I think this kind of rewrite is beyond my skill level.

What is the question / expectation here?

@pailhead I’m sorry for the vagueness here.

I am trying to create a scene where the models are warped in a particular way.

Running the code as it is–and even with just the shaders used in Unity without the additional code changes–nothing is appearing on the screen, but there are no errors in my console.

I don’t have a particular expectation from you all, but I would really appreciate it if someone more well-versed in WebGL/THREE were able to identify some obvious issues with my code, or could suggest a different way of approaching my problem.

Well the obvious issue is naming things. temporaryVariable_17 is a pretty bad name and it’s really hard to tell what it does. My advice would be to study the code and understand what is going on. Then look into which parts need to change on three.js side. There’s a whole bunch of things that can fail here.

@pailhead Thanks for your response, and I’m inclined to agree.

The variable names etc. come from the fact that it is (I believe) computer-generated code. I was hoping I might be able to import the shaders as-is because the mathematics behind them are beyond me recreating from scratch. I’ll try digging some more through the Unity project and seeing if I can figure out where these are all coming from (and what might just be useless project junk) and what does what, but it may very well be a lost cause.

Thanks for your time!

There are ways to go about this, for example test the vertex shader first to see if you’re getting the correct attributes. Then pass the varyings then finally test the fragment computation. But id start with something more simple and legible.