So what I ended up doing was plotting RGB points in 3d space and using Delaunay triangulation with delaunator library and example from this thread. So RGB cube idea worked out in part, but I could not figure out how to deal with empty space produced by surface deformation, so triangulation was a way to go. In the end it worked out pretty well in my opinion
Hope this helps anyone who finds this thread in future