Rendering pictures captured by the camera as texture react native and react three fiber

Questions
1

Hello, I have been struggling with this one for a while. I want to capture images by the camera using expo camera and render them as texture and so far all I am getting is a black texture or an error. here is the current code

 const captureImage = async () => {
    if (cameraRef.current) {
      const photo = await cameraRef.current.takePictureAsync({
        base64: true,
        shutterSound: false,
        skipProcessing: true,
      });
   
      return photo?.uri;
    }

function ImageT({ src }: { src: string }) {
  try {
    const texture = useLoader(TextureLoader, src as string);
    console.log(texture);

    return (
      <mesh>
        <planeGeometry attach="geometry" args={[2, 2]} />
        <meshBasicMaterial
          side={THREE.DoubleSide}
          attach="material"
          map={texture}
        />
      </mesh>
    );
  } catch (e) {
    console.log(e);
    return null;
  }
}
2

You need to share more code, there’s too much missing to have an idea of what’s going wrong. ideally in a live example on codepen / jsfiddle.

For debugging, I would also add the generated image to the DOM, so you can figure out if the problem is with taking the picture, or with rendering it in your 3D scene.

3

This is almost all the logic but here is the rest

the App component:

export default function App() {
  const { width, height } = Dimensions.get("window");
  const { cameraRef, points, ...rest } = useCaptureImage();

  return (
    <>
      <Canvas
        camera={{ position: [0, 0, 0], fov: 90, aspect: width / height }}
        style={{
          position: "absolute",
          left: 0,
          top: 0,
          right: 0,
          bottom: 0,
        }}
      >
        <PanoramaCapture points={points} cameraRef={cameraRef} {...rest} />
      </Canvas>

      <CameraView
        animateShutter={false}
        style={{
          flex: 1,
          zIndex: -1,
        }}
        ref={cameraRef}
      />
    </>
  );
}

here is the panorama capture component:

function PanoramaCapture(props: {
  points: PhotoPoint[];
  targetPoint: THREE.Vector3 | undefined;
  handleCapture: (camera: CameraType) => Promise<void>;
  cameraRef: React.RefObject<CameraView>;
}) {
  // const [capturedPoints, setCapturedPoints] = useState<THREE.Vector3[]>([]);
  // const [nextPoint, setNextPoint] = useState<THREE.Vector3>();
  const sensor = useAnimatedSensor(SensorType.ROTATION, { interval: "auto" });

  const [isReadyToCapture, setIsReadyToCapture] = useState(false);
  const CAPTURE_ANGLE = 30 * (Math.PI / 180); // 30° in radians
  const { camera } = useThree();

  const { points, targetPoint, handleCapture } = props;

  // const { getPoints } = useGetTargetPosition();

  const { getCameraVectors } = useRotateObject({ object: camera });

  useFrame(({ camera }) => {
    const { forward, up } = getCameraVectors();

    // Option 1: Look in the direction
    const target = new THREE.Vector3().copy(camera.position).add(forward);
    camera.up.copy(up);
    camera.lookAt(target);
  });

  return (
    <>
      <axesHelper position={[0, 0, -1]} args={[5]} />
      <mesh
        onClick={async () => {
          await handleCapture(camera);
        }}
        position={[0, 0, 0]}
      >
        <sphereGeometry args={[5, 32, 16]} />
        <meshBasicMaterial
          color="#ffff00"
          side={THREE.DoubleSide}
          transparent={true}
          opacity={0.8}
        />
      </mesh>
 

      {points.map((p, i) => (
        <Suspense key={i}>
          <group position={p.position}>
            {p.imageUri && <ImageT src={p.imageUri} />}
          </group>
        </Suspense>
      ))}

    </>
  );
}

we basically capture the point which trigger an image capture and pass the image uri to the Image component to render it as texture but it displays a black texture, we can even try to pass the base64 string and it will still not work.

PS: if I pass the image uri to the normal react native image component the image will be displayed so there is no problem with the image.

1 Like
5

Ah I didn’t realise it was React Native.

Since there’s no problem with the image the issue could be with your app’s react lifecycle.

What does the console show when you log the texture? What does it show if you try to load the src prop passed to the ImageT function?

If you pass that texture a base 64 string that you’ve created beforehand, or an external URL, does it work? eg
const texture = useLoader(TextureLoader, aBase64String);
const texture = useLoader(TextureLoader, anImageUrl);

If it does, then that means you need to look into your state / props management.

6

if I use an external url(ex: http) then it will work fine, but for the following:

  • const texture = useLoader(TextureLoader, aBase64String); → here it will log the base 64 string and fails for loading the texture

  • const texture = useLoader(TextureLoader, anImageUrl); here the url is file:///data/user/0/host.exp.exponent/cache/ExperienceData/%2540anonymous%252F360-capture-a82304ee-133c-42ea-8bb5-06e9b5744639/Camera/eb0d28d8-0a4c-4983-bc5b-39bc0716189e.jpg
    and the texture object is: {"anisotropy": 1, "center": [0, 0], "channel": 0, "colorSpace": "srgb", "flipY": true, "format": 1023, "generateMipmaps": true, "image": "ef3b4eaa-21a3-489b-bc84-44d58616afd1", "internalFormat": null, "magFilter": 1006, "mapping": 300, "metadata": {"generator": "Texture.toJSON", "type": "Texture", "version": 4.6}, "minFilter": 1008, "name": "", "offset": [0, 0], "premultiplyAlpha": false, "repeat": [1, 1], "rotation": 0, "type": 1009, "unpackAlignment": 4, "uuid": "0c2c1424-69f0-45d3-aad0-bd43c460cc08", "wrap": [1001, 1001]}
    this is also logged in the console: (NOBRIDGE) LOG EXGL: gl.pixelStorei() doesn't support this parameter yet!

and the output on the screen is going to be a black texture