What is different about a VR/AR multiplayer backend?

The durable backend responsibilities are familiar: identity, entitlements, persistent state, sessions, moderation and server allocation. The difference is the realtime data plane: head/hand transforms, voice and spatial metadata are higher-frequency and should normally travel through the networking/session layer rather than a REST database API.

Should the backend store every hand or head pose?

Usually no. Pose streams are transient realtime state. Persist only what the product needs across sessions: world objects, anchor identifiers or metadata, calibration/configuration and audit/support information. Raw sensor streams should have an explicit product and retention reason before you store them.

How should I think about VR latency?

Keep local rendering/motion-to-photon latency separate from network latency for remote avatars and authoritative simulation. The backend cannot fix a slow local render loop. Measure RTT, jitter, packet loss, interpolation error and server tick delay independently and set budgets from your own interaction design.

Do spatial anchors use one universal backend format?

No. Anchor capabilities and persistence differ by platform and SDK. Treat the platform anchor identifier/blob and your game-world metadata as separate data. For cross-device experiences, define your own world/session coordinate contract and validate what each target platform can actually resolve.

VR/AR Multiplayer Backend: Presence, State & Sessions

Backend architecture for multiplayer VR and AR: identity, shared presence, persistent state, session allocation, spatial metadata and realtime boundaries.

Written and reviewed by Supercraft Engineering

VR and AR do not require a completely different backend. They require a clearer split between durable game state and a high-frequency realtime presence stream. Identity, progression, entitlements, sessions and world persistence belong in the ordinary backend. Head/hand poses, voice and moment-to-moment transforms belong in the networking layer.

Do not send pose updates through your player-data API. The backend should help players find/join a session and persist what survives it; the session transport should carry transient presence.

1. Separate the Durable and Realtime Planes

Data	Where it belongs	Why
Identity, entitlements, inventory	Persistent backend	Transactional and durable
Session membership / join authorization	Backend + session service	Security and lifecycle
Head / hand transforms	Realtime transport	Frequent and disposable
Voice	Voice/realtime service	Streaming media, not document state
World objects that survive sessions	Persistent backend / object storage	Needs versioning and recovery
Spatial anchor references	Backend metadata + platform anchor system	Platform-specific resolution plus game ownership

2. Spatial Anchors: Store the Contract, Not Assumptions

Anchor APIs differ across Meta, Apple, Google and other XR stacks. Instead of pretending they share one persistence/range model, keep a neutral game record around the platform object:

{
  "world_id": "world_123",
  "anchor_provider": "platform-specific",
  "anchor_ref": "opaque-provider-id",
  "game_transform": { "position": [0,0,0], "rotation": [0,0,0,1] },
  "created_by": "player_123",
  "schema_version": 2
}

The client/platform SDK resolves the opaque anchor. Your backend controls who can discover it, what game object it corresponds to, and how migrations/versioning work.

3. Realtime Presence: Budget from Interaction, Then Measure

There is no universal “VR backend must be under N milliseconds” number. A local hand touching a local object has a different budget from a remote avatar gesture or an authoritative projectile. Measure:

network RTT, jitter and packet loss between peers/server;
simulation tick and queueing delay on the authoritative process;
interpolation/prediction error visible on remote avatars;
voice delay separately from gameplay state;
local render/motion-to-photon latency with platform profiling tools.

Choose transport, tick/update rate and regional placement from those measurements instead of copying a single latency target from another title.

4. Bandwidth: Calculate Your Own Pose Packet

A useful worked calculation is simple and auditable. If one pose update contains head position+rotation plus two hand position+rotation transforms, serialize a representative packet and measure its actual bytes after your networking library adds headers. Multiply by update rate and players visible to each receiver. Then test interest management, quantization and delta compression.

This is more reliable than tables claiming fixed “KB/s for VR,” because skeleton size, finger tracking, compression and visibility differ dramatically by game.

5. Authority Still Matters

High-frequency presence can be client-produced without making valuable state client-authoritative. A client may tell the server where its tracked hand is; the server still decides whether that hand can pick up a scarce item, deal damage, spend currency or alter persistent world state.

Keep exploit-sensitive actions as commands validated against session/world rules rather than trusting a streamed transform as proof that an action was legal.

6. Privacy: Treat Spatial/Sensor Data as Sensitive by Design

Room geometry, voice, gaze and body-motion data can reveal more than ordinary game telemetry. Collect only what the feature needs, separate ephemeral transport data from persisted analytics, document retention, and make deletion/export workflows aware of any stored sensor-derived records. For regulatory design, see the privacy backend guide.

7. Session Architecture

player authenticates
  -> backend returns join authorization
  -> match/session service chooses a realtime instance
  -> client connects to realtime transport
  -> transient pose/voice stays in session plane
  -> authoritative events update durable backend at checkpoints/end
  -> session closes; durable state survives

This separation also makes scaling saner: session capacity can grow/shrink with concurrency while persistent state remains independent.

8. Cost Model

Do not price XR from a “1,000 concurrent VR users = $X” table. Model the components you actually use: occupied realtime server hours, relay/voice traffic, egress, anchor/object storage, persistence calls, observability and any media processing. Passthrough video or cloud rendering is a separate media workload and can dominate cost; many multiplayer XR games do not need it at all.

Practical Roadmap

Make the interaction work locally and identify which state must be authoritative.
Add authenticated sessions and a realtime transport for transient presence.
Persist only the world/player state that must survive disconnects.
Add platform anchors where the experience actually needs physical-space persistence.
Benchmark representative devices/networks before adding more regions.
Audit stored sensor/spatial data and remove anything without a clear purpose.