Apple Vision Pro Update: Video delivers

In summer 2023 I shared my enthusiasm and anticipation about the new Apple Vision Pro headset, specifically the potential implications for video.

A few months later, a report from The Verge lent credence to the idea that the video we shoot and consume could be going up a notch.

The reviewer, Lance Ulanoff, experienced video and stills shot using a capture mode on the latest iPhone that maps well into the AVP virtual space. And that’s with two lens, not the array of on the headset. Apple knows what they have here for immersive video, and while their “spatial computing” messaging emphasizes the broader utility of the their system, I believe immersive video is a big selling point. Ulanoff:

“[While] I did not feel as if I was inside any of them, each one felt more real, and whatever emotions I had watching them were heightened. I suspect that when consumers start experiencing the Vision Pro and spatial videos for themselves they might be surprised at the level of emotion they experience from family videos – it can be quite intense.”

It’s worth noting the term “spatial video.” It’s Apple’s name for an actual file format/encoding. “Immersive video” goes a step further, as explained really well here .

That’s probably becoming the term of art, not “immersive capture” but I’ll hold onto it a bit longer. I think spatial audio and spatial video and immersive video combined represent the form.

It appears that at least initially, the AVP video capture and playback is limited to a more conventional rectangular or square screen floating in the field of view (but stereoscopic.) For playback, the AVP can extend and blur the edges of the screen using ‘foveated rendering,’ a technique that tracks eye movement and sort of de-rezzes what’s in your peripheral vision.

I thought it would be a matter of time and progress in compute & storage efficiency until they allow all the cameras to collect and record at once and extend the field of view with more sharpness. I was wrong. Immersive Video, another Apple term and format that very similar to VR180, does exactly that. You can move your head around in the virtual space.

I’ve seen [mostly] professional reviewers on Youtube demonstrating the AVP capabilities, like the excellent channel, Spatial Video Insights, (not just Brownlee, Neistat, et al. I need to dive back in and see what people are doing with it in the way of immersive capture for storytelling. The audience is very small at this point, but with the number of units out there now, there are already people adapting existing content and experimenting with storytelling. Can hardly wait!