Gestures & Interactions

Gestures come in two layers:

• Object gestures – drag, rotate, or resize a specific object. Enable them on the object, then optionally react in a script.
• Screen gestures – raw pan / pinch / rotate anywhere on screen, independent of any object. The building block for custom mechanics like flick-to-launch.

Screen gestures fire on iPhone, iPad and Mac – not Apple Vision Pro, where input is spatial. Gate a portable script with environment.features.has('screenGestures') and fall back to UI controls. See Feature Detection.

Object gestures

Enable a gesture on the descriptor (or in the editor), and the user can manipulate the object directly:

var crate = await scene.createEntity(
    createBox(0.2, 0.2, 0.2)
        .name('crate')
        .gestures({ drag: true, rotate: true })
);

React to the live manipulation with on('gesture') – it fires continuously while the user moves the object. Read e.phase ('began' / 'changed' / 'ended') and, by kind, e.translation / e.rotation / e.scale:

crate.on('gesture', function(e) {
    if (e.phase === 'ended') {
        console.log('moved by', e.translation.toString());
    }
});

doubleTap and longPress are discrete object gestures, available in code:

crate.on('doubleTap', function() { /* ... */ });
crate.on('longPress', { minimumDuration: 0.5 }, function() { /* ... */ });

Release behaviour

When a dragged object is let go, releaseBehavior decides what happens next:

createSphere(0.04)
    .physics({ mode: 'dynamic', mass: 0.2 })
    .gestures({ drag: true, releaseBehavior: 'momentum' });

releaseBehavior	On release
'stay' (default)	Stays where it was let go, then falls under gravity if it has a dynamic body.
'reset'	Animates back to where the gesture started.
'momentum'	Keeps the motion the gesture gave it – a flick throws it, a slow release lets it drift. Tune with momentumScale / angularMomentumScale.

'momentum' gives you a throw with no manual velocity maths – the engine measures the release motion for you. Reach for the scripted approach below only when you need finer control.

Screen gestures

scene.on('pan' | 'pinch' | 'rotate') fires for any such gesture anywhere on screen. Read e.position (normalized), e.phase, and by kind translation / scale / rotation plus velocity:

// Flick strength from a swipe in the bottom of the screen.
scene.on('pan', { area: [0, 0.6, 1, 0.4], phase: 'ended' }, function(e) {
    console.log('flick speed', e.velocity.length());
});

Outside AR (the desktop preview, the Mac app), a one-finger pan also orbits the camera by default – so a pan your script reacts to would move your object and swing the camera at once. consume claims the gesture for your script alone: return true from its predicate (evaluated as the gesture begins) and the camera ignores that gesture. Omit consume for the smart default, which claims a gesture automatically when it begins on a scene object:

// Claim the pan only when it starts on the ball; every other pan still orbits the camera.
scene.on('pan', { consume: function(e) { return e.hits(ball); } }, function(e) { /* ... */ });

See scene.on for the full payload and consume options.

Composing a custom mechanic

Specific mechanics – pickup-and-throw, drag-to-aim, swipe-to-navigate – aren't engine features; you compose them in a script from the primitives above plus physics. A flick-to-launch, in miniature:

var ball = scene.findEntity({ name: 'ball' });

// Claim a pan that starts on the ball; on release, convert its screen velocity to an impulse.
scene.on('pan', { consume: function(e) { return e.hits(ball); } }, function(e) {
    if (e.phase !== 'ended') { return; }
    var launchScale = 0.002;  // screen points/sec -> world impulse; tune for the feel you want
    var lift = 0.4;           // upward component so the throw arcs
    ball.applyImpulse(new Vector3(e.velocity.x * launchScale, lift, -e.velocity.y * launchScale));
});

For a full worked example (velocity smoothing over the tail of the drag, scoring, reset), the same idea scales up – measure the release velocity, apply it as an impulse, and let physics carry the object.