How can I achieve pointerEvent all do not affect raycaster in react-three-fiber?

Questions
1

The raycaster work fine when make pointerEvent to none, but I need listen to some event like click on html tag that I change its pointerEvnet to all

Then the coordinates of boxGeometry following mouse will be misaligned if using pointerEvent all

my demo code like

import React, { useRef } from 'react'
import ReactDOM from 'react-dom/client'
import { Canvas, useFrame } from '@react-three/fiber'
import './styles.css'
import { Html } from '@react-three/drei'

function Thing() {
  const ref = useRef()
  const planeMeshRef = useRef()
  useFrame(({ camera, mouse, raycaster }) => {
    raycaster.setFromCamera(mouse, camera)
    if (planeMeshRef.current) {
      const intersects = raycaster.intersectObjects([planeMeshRef.current])
      if (intersects[0]) {
        let p = intersects[0].point
        if (ref.current) {
          ref.current.position.copy(p)
        }
      }
    }
  })

  const handleClick = () => {
    console.log('clicked')
  }
  return (
    <>
      <Html wrapperClass="topWords">
        <div className="wrapper">
          <p
            style={{
              // The coordinates of boxGeometry following mouse will be misaligned if using pointerEvent all
              pointerEvents: 'all'
            }}
            onClick={handleClick}>
            hello world
          </p>
        </div>
      </Html>
      <mesh ref={ref}>
        <boxGeometry attach="geometry" args={[1, 1, 1]} />
        <meshNormalMaterial attach="material" />
      </mesh>
      <mesh ref={planeMeshRef}>
        <planeGeometry args={[1000, 1000]}></planeGeometry>

        <meshBasicMaterial transparent />
      </mesh>
    </>
  )
}

const root = ReactDOM.createRoot(document.querySelector('#root'))

root.render(
  <Canvas>
    <Thing />
  </Canvas>
)

and online demo

You can see the coordinates of boxGeometry is wrong when mouse hover ‘hello world’

2

/cc

3

Since I didn’t get an answer so I asked questions in many places to improve the answer rate