Seeking a robust multi-touch pattern for a keyboard with multi-key presses per finger

[DanielAtCosmicDNA]

Hello everyone,

I’m developing a virtual stenography keyboard using React Three Fiber and I’ve hit a wall with a tricky multi-touch issue. I’m hoping to get some advice from the community on how to build a more reliable solution.

The main challenge is that a single finger (one pointerId) might be large enough to press multiple keys at once. The keyboard needs to track all keys under all fingers and then register the final “chord” when the user lifts their fingers off the screen.

You can see a live version here: And the source code is here.

Current Implementation

My approach is to give each individual key its own gesture handler.

1. Component Structure: The keyboard is composed of many small Hexagon meshes. The main StenoKeyboard component manages the overall state.

2. State Management: In StenoKeyboard, I use a Map to track the state of all active pointers:

const [pressedKeys, setPressedKeys] = useState(new Map());

The Map’s keys are the pointerId for each touch, and the values are a Set of the keyIds being pressed by that specific finger.

3. Per-Key Event Handling: This is the crucial part. Each individual Hexagon component has its own useDrag gesture handler (from @use-gesture/react via a custom hook).

When a finger touches down or drags over a hexagon, that hexagon’s useDrag handler fires.
It then calls a shared updatePressedKeys function to add its own keyId to the Set associated with the current pointerId.
When the finger is lifted (last: true in the gesture state), it clears the Set for that pointerId.
Here’s a look at the Hexagon component:

// In components/Hexagon.js
const Hexagon = ({ geometry, name, pressedKeys, updatePressedKeys, ...props }) => {
  const keyId = name;
  // This custom hook wraps a useDrag handler
  const dragProps = useDrag({ keyId, pressedKeys, updatePressedKeys });

  return (
    <group {...dragProps} {...props}>
      <mesh userData={{ keyId }}>
        {/* ... material and geometry */}
      </mesh>
    </group>
  );
};

The rationale for this design is to support multi-key presses from a single finger. If a finger is large enough to touch two hexagons at once, both of their useDrag handlers should fire for the same pointer event, adding both of their keyIds to the state.

The Problem: “Stuck” Keys

The keyboard works, but it’s not reliable. Frequently, keys get “stuck” in the pressed state.

My theory is that with so many adjacent event handlers, I’m running into race conditions or dropped events. For example, if a finger lifts up precisely on the boundary between two hexagons, or moves very quickly, one of the hexagons might miss the pointerup or pointerleave event. Its useDrag handler never gets the last: true signal, so it never clears its keyId from the state map, leaving the key “stuck.”

The Core Question

How can I reliably track multiple keys per finger without the fragility of per-key event listeners?

1. Is there a better pattern? A centralized event handler on a single large plane seems more robust for tracking pointers, but a single raycast from that handler would only detect one key at a time. How could a centralized approach be adapted to find all keys within a certain radius of the pointer’s location?
2. Improving the current approach: If I stick with per-key listeners, are there techniques to make the state cleanup more foolproof, ensuring a pointerup event anywhere on the screen correctly cleans up the state for its pointerId?
3. Alternative Ideas: Are there other drei helpers or three.js features that could solve this more elegantly?

I feel like I’m fighting the framework a bit here and would be grateful for any insights or suggestions on a better architecture. Thank you!

[trueshko]

I see you problem is that hexagons are overlapped. Even if you click the mouse between the keys, both are pressed.

[trueshko]

I’m not very familiar with React, but I know that Fiber can catch events directly on each mesh. I think this method would be much more accurate.

[DanielAtCosmicDNA]

In a stenography keyboard the ability to press multiple keys with a single finger is an important feature. For example, as there is no “I” character, it is represented by the combination of “E” and “U” pressed together. Additionally I wanted to have the ability to drop a key when dragging out of it. Or similarly, to ultimately allow a piano slide.

The behaviour you noticed happens because I implemented multiple raycasters per touch (5 at the moment) to simulate a 2d finger press touchscreen-stenography-keyboard/src/components/hooks/useDrag.js at main · CosmicDNA/touchscreen-stenography-keyboard.

[manthrax]

Generally I feel like this would be optimally handled closer to a pointerdown event handler itself, and then set the states of the visual representation in response, in your graphics loop.. like maintain an array of pressed buttons.. and their position in screen space.. then on the pointerdown/pointermove, you quickly compare to the current screen positions, and set pressed/not pressed on those states. Then let the rendering layer react to it next frame. This is much faster than raycasting (just checking pointer position->distance to->hex screen position.. especially for a lot of keys.. and doesn’t restrict you to the hex boundary, rather you can find the closest buttons within a certain distance of each hit. But maybe that’s overkill, idk.

[Lawrence3DPK]

Could it be an idea to place invisible “overlap” mesh boxes between any two keys that could be pressed at the same time, these could extent the full width of the keys and overlap something like a quarter or third of each key pairs in breadth, if this theoretical overlap mesh is intersected it would mean both keys are pressed if not only the individual intersected key would register as pressed? Just an idea..

[DanielAtCosmicDNA]

Hello everyone,

Thank you all for the input on this topic. After a lot of experimentation and deeply appreciating each and everyone of your suggestions, I’ve landed on a robust pattern for handling multi-touch gestures that I wanted to share, as it might be useful for others building complex interactive experiences like virtual keyboards.

The Journey

My initial goal was to have a system that could track multiple simultaneous finger drags, with each finger potentially pressing multiple keys at once (via raycasting a “fingerprint” area).

1. Initial Library (@use-gesture/react): While excellent for single-pointer gestures, its useDrag hook is not designed for multi-touch drag out of the box.
2. Gesture Library (Hammer.js): I then tried Hammer.js, which has powerful multi-touch capabilities. However, this led to a lot of complexity in handling the different gesture recognizers (pan, tap) and dealing with race conditions between them and the raw hammer.input events. It became difficult to reliably distinguish a “tap” from the start of a “drag.”
3. The Breakthrough (Native Pointer Events): The final and most robust solution was to remove the gesture library dependency entirely and use the browser’s native Pointer Events API. This approach is dependency-free, highly performant, and gives you the precise control needed for this kind of interaction.

The Final Pattern: A Custom Hook + Consumer

The pattern consists of two parts: a custom React hook (useMultiTouchDrag) that abstracts away all the pointer logic, and a component (KeyPressDetectionFloor) that consumes the clean event stream from the hook.

1. The useMultiTouchDrag Hook

This hook is the heart of the solution. Its only job is to track individual pointers and emit a clean, predictable event stream.

Responsibilities:

• Attaches native pointerdown, pointermove, pointerup, and pointercancel listeners to the canvas.
• Uses element.setPointerCapture() to ensure events are tracked even if a finger moves off-canvas.
• Maintains a pool of stable, reusable fingerIds (e.g., 0-9). A new touch gets the lowest available ID, and the ID is returned to the pool on release.
• Emits a simple, structured event: { type: 'onDragStart' | 'onDragMove' | 'onDragEnd', fingerId: number, pointer: PointerEvent }.

Here is the final code for the hook:

// hooks/use-multi-touch-drag.js
import { useEffect, useRef } from 'react'
import { useThree } from '@react-three/fiber'

/**
 * A hook to handle multi-touch drag gestures on the R3F canvas,
 * emitting clean onDragStart, onDragMove, and onDragEnd events for each touch.
 * @param {Function} handler - The function to call on drag events.
 */
const useMultiTouchDrag = (handler) => {
  const { gl } = useThree()
  const trackedPointers = useRef(new Map())
  const fingerIdPool = useRef(Array.from({ length: 10 }, (_, i) => i)) // Pool of IDs 0-9

  useEffect(() => {
    const element = gl.domElement
    if (!element) return

    const handlePointerDown = (event) => {
      element.setPointerCapture(event.pointerId)
      if (fingerIdPool.current.length > 0) {
        const fingerId = fingerIdPool.current.shift()
        trackedPointers.current.set(event.pointerId, { fingerId, pointer: event })
        handler({ type: 'onDragStart', fingerId, pointer: event })
      }
    }

    const handlePointerMove = (event) => {
      const tracked = trackedPointers.current.get(event.pointerId)
      if (tracked) {
        tracked.pointer = event
        handler({ type: 'onDragMove', fingerId: tracked.fingerId, pointer: event })
      }
    }

    const handlePointerUp = (event) => {
      element.releasePointerCapture(event.pointerId)
      const tracked = trackedPointers.current.get(event.pointerId)
      if (tracked) {
        const { fingerId } = tracked
        handler({ type: 'onDragEnd', fingerId, pointer: event })
        trackedPointers.current.delete(event.pointerId)
        fingerIdPool.current.push(fingerId)
        fingerIdPool.current.sort((a, b) => a - b)
      }
    }

    element.addEventListener('pointerdown', handlePointerDown)
    element.addEventListener('pointermove', handlePointerMove)
    element.addEventListener('pointerup', handlePointerUp)
    element.addEventListener('pointercancel', handlePointerUp) // Treat cancel like up

    return () => {
      element.removeEventListener('pointerdown', handlePointerDown)
      element.removeEventListener('pointermove', handlePointerMove)
      element.removeEventListener('pointerup', handlePointerUp)
      element.removeEventListener('pointercancel', handlePointerUp)
    }
  }, [handler, gl.domElement])
}

export default useMultiTouchDrag

2. The Consuming Component

This component uses the hook and focuses purely on the application logic (raycasting and updating state). It doesn’t need to know anything about pointers, gestures, or race conditions.

Responsibilities:

• Calls useMultiTouchDrag with a handler.
• Uses a simple switch statement to react to onDragStart, onDragMove, and onDragEnd.
• Performs raycasting based on the pointer data in the event.
• Updates the application’s global state (pressedKeys Map) on a per-fingerId basis.

Here is the code for the consumer:

// components/KeyPressDetectionFloor.js
import PropTypes from 'prop-types'
import React, { useRef } from 'react'
import { getCircularPoints, eqSet } from './utils/tools'
import { Vector2, Raycaster } from 'three'
import useMultiTouchDrag from './hooks/use-multi-touch-drag'
import { useThree } from '@react-three/fiber'

const fingerResolution = 5
const rawFingerModel = getCircularPoints(fingerResolution, fingerResolution, 0.05)
const refFingerVectors = rawFingerModel.map(v => new Vector2(...v))

const KeyPressDetectionFloor = ({ pressedKeys, updatePressedKeys, ...props }) => {
  const groupRef = useRef()
  const { camera, size: canvasSize } = useThree()

  const setFingerPressedKeys = (fingerId, newSet) => updatePressedKeys((map) => map.set(fingerId, newSet))
  const clearFingerPressedKeys = (fingerId) => updatePressedKeys((map) => map.delete(fingerId))

  useMultiTouchDrag((ev) => {
    const processDrag = () => {
      const keyMeshes = groupRef.current?.parent?.children
        .filter(c => c.name === 'key group')
        .map(g => g.children[0])
        .map(g => g.children[0])

      if (!keyMeshes) return

      const { clientX, clientY } = ev.pointer
      const coords = new Vector2(
        (clientX / canvasSize.width) * 2 - 1,
        -(clientY / canvasSize.height) * 2 + 1
      )
      const fingerVectors = refFingerVectors.map(v => v.clone().add(coords))

      const intersectedKeys = new Set([...fingerVectors.flatMap(v => {
        const raycaster = new Raycaster()
        raycaster.far = 50
        raycaster.setFromCamera(v, camera)
        return raycaster.intersectObjects(keyMeshes)
      }).map(o => o.object.userData.keyId)])

      const previousSet = pressedKeys?.get(ev.fingerId)
      if (!eqSet(previousSet, intersectedKeys)) setFingerPressedKeys(ev.fingerId, intersectedKeys)
    }

    switch (ev.type) {
      case 'onDragEnd':
        clearFingerPressedKeys(ev.fingerId)
        break
      case 'onDragStart':
      case 'onDragMove':
        processDrag()
        break
    }
  })

  return (
    <group ref={groupRef} {...props} />
  )
}

KeyPressDetectionFloor.propTypes = {
  pressedKeys: PropTypes.instanceOf(Map),
  updatePressedKeys: PropTypes.func
}

export default KeyPressDetectionFloor

This separation of concerns has proven to be extremely effective. The hook is reusable and robust, and the component logic is clean, declarative, and easy to maintain.

I hope this detailed breakdown is helpful to anyone else tackling similar multi-touch challenges in three.js and React. this solution was committed to the main branch of the repo and is now available in the project’s website.