Apple is opening up what is possible on Apple Vision Pro by implementing physical object tracking and expanded spatial accessory support. Here’s what that means for users when visionOS 27 launches.
When Apple initially launched Apple Vision Pro, it was very proud of the hand and eye tracking it had developed. Controllers weren’t even mentioned beyond PlayStation’s standard one for playing 2D iPad games.
Slowly, it seems Apple realized that having zero haptic or physical feedback in games and interactions would severely impact what was possible. visionOS 26 introduced spatial accessories, and now visionOS 27 is expanding what is possible, both with fully realized controllers and basic objects.
First, it is important to understand the difference between a spatial accessory and an interactable object. Spatial accessories, like PSVR2 controllers or the Logitech Muse pen, pair over Bluetooth and might have buttons, haptics, and other features.
Anything else being added to visionOS is an object. It can be tracked using two methods — 3D model reference and LED tracking or visual markers.
Here’s the 15-minute developer video Apple shared on the subject.
Basically, a 3D render of an object can make Apple Vision Pro aware of that object through software. Added sensors and chips can make an object into a full spatial accessory.
Spatial accessories
When you think of a VR controller, you’re probably thinking of the Meta Quest ones that wrap around your hand. The PSVR2 controllers are similarly designed and work with Apple Vision Pro.
A 3D render of a spatial accessory flashlight with LED markers
However, a spatial accessory doesn’t necessarily have to be a controller. Apple defines a spatial accessory as having three properties:
- LED constellation for tracking
- Internal Measurement Unit (IMU) chip
- Bluetooth
Inputs and outputs, like buttons and haptics, are possible, but not required.
The demo focuses on a flashlight accessory, for example. It has trackable LEDs, an on/off button, and a Bluetooth connection to Apple Vision Pro.
By integrating a 3D model of this flashlight into the accessory developer’s app, Apple Vision Pro can easily track it thanks to the accurate model and LED constellation. Also, spatial accessories and tracked objects added via apps are addressable system-wide, though they won’t get all the benefits of being a full-fledged game controller like PSVR2.
While this doesn’t necessarily mean we’ll see previous generations of VR controllers come to Apple Vision Pro, it does mean it’ll be easier for accessory makers to build new ones. It may also be possible to get existing accessories to work, but in an unofficial way unless done by the manufacturer itself.
Spatial accessories can be partially occluded and still be tracked with low latency. The high frequency tracking feature allows tracking up to the refresh rate of the Apple Vision Pro.
Trackable objects
visionOS 27 also enables the ability for developers to add object tracking via object recognition or attached tracking fobs. It’s a step down from a spatial accessory, as it doesn’t require any additional technology to be added to the object being tracked.
A scan of a physical object like a globe can bring it into software via ARKit
The example here is a globe. Since object tracking is also coming to iPhone via ARKit, it shows a UI overlaid on the physical globe that spins without lag when the physical globe is spun.
This is accomplished via using an accurate 3D model in the app to track the object, though not at as high a frequency due to the lack of LED markers. More advanced tracking is possible by attaching a third-party fob though.
The video references fobs from DFRobot and MIKROE that can help anchor physical objects in spatial environments. The use case demoed here was a fob on a steering wheel.
By attaching a fob to a standard steering wheel rig, the app was able to overlay a full 3D vehicle around it. This would be an ideal use case when playing racing simulators like iRacing.
A preview of what’s to come
The Apple Vision Pro is a niche product that’s too heavy and expensive for many users to adopt. However, its existence enables Apple to further develop its visionOS platform.
Apple Vision Pro exists as a preview of what will eventually be possible in AR glasses
While the technology is still in its early stages, it is clear with each update that Apple expects wider adoption eventually. The ability to track real-world objects is key when designing AR interactions.
While true AR glasses, or Apple Glass as we’ve called them, might not arrive until 2029 or later, the groundwork is being laid today.
I could easily imagine my smart water bottle from Hidratespark being turned into an object for Apple Vision Pro. Perhaps just something as simple as making it visible in fully immersed environments, or showing a water-consumed metric when looking at it would be enough.
Hidratespark wouldn’t need anything they don’t already have to make it happen either. Technically, a third-party water tracker could do this if they created 3D models of popular water bottles.
Another concept I could think of is that the developer of Plant Daddy could get 3D models of certain kinds of flower pots and turn them into objects recognized by Apple Vision Pro. The pots themselves could become physical anchors that show plant data when you look at them directly.
It’s a really interesting technology, but it will require developers to actually utilize it. As always, that’s the biggest problem with Apple Vision Pro, and one Apple didn’t directly address with visionOS 27.
Stephan is the sports journalist for the Maple Grove Report.
