Open Sourcing ImmersiveWatchParty
(Open Source, visionOS, RealityKit) Why I'm open-sourcing the SharePlay co-watching SDK, and what's still missing from Apple's native APIs.
Today, I'm open-sourcing the ImmersiveWatchParty SDK, a project I started in November 2024.
The original idea was simple: immersive video is already compelling on its own, but how much better would it be to watch it with friends? Sports, music, concerts, comedy, plays, whatever you're into.
Where it started
I started building a feature that lets people call each other on FaceTime with their Spatial Personas and watch the same immersive video together. It took a lot of trial and error to build and test, especially because it required testing with other visionOS users, and the visionOS community, while small, is tight-knit.
For me, the thought of sitting cage-side at a UFC event with my friends, reacting to every technique and knockout blow, is what propelled me to build Vantage, my own immersive combat sports platform. With modern mixed-reality devices, that's actually becoming possible now. But I found that pulling it off takes more than just a video player, it takes a dedicated synchronization engine. So I built one, and with help from a lot of friends, got the co-watching feature working inside Vantage.
After integrating the co-watching feature into Vantage, I decided it would be worth extracting it into its own standalone SDK and Swift package. That's what I'm releasing today.
The gap in Apple's native APIs
Along the way I ran into a real limitation in Apple's own frameworks: the native video player APIs can't render Spatial Personas side by side while watching immersive content, even with SharePlay active. Oddly, Personas can sit side by side and watch a regular 2D movie together, even inside an immersive movie theater scene. Apple just hasn't shipped the ability to sit side by side while watching stereoscopic video.
OpenImmersive, maintained by Anthony Maës and Acute Immersive and derived from Spatial Player by Mike Swanson, turned out to be the key. Spatial Player was the original open-source project built on RealityKit to render stereoscopic VR180 video, back when visionOS 1.0 had no native immersive video player API at all.
Apple eventually shipped its own high-level AVPlayer-based implementation with native immersive video support in a later visionOS release. When I upgraded my own app to use it, I ran into a compatibility flaw: it doesn't let Spatial Personas sit side by side while watching immersive video. The native player has real upsides too, easier to work with, more robust, backed by Apple's own development standards, and smoother with live HLS streaming, but that one flaw was a dealbreaker. So I stayed on the RealityKit-based implementation and kept building on top of it.
That's the honest trade-off if you want to integrate this into your own app: you have to give up Apple's native video player APIs, because they're too high-level to link Spatial Personas side by side into the same scene. Until Apple fixes that, this is the workaround.
Beyond video sync
Even if Apple eventually fixes that gap, I don't think they'll ship everything I've built into this SDK anytime soon, namely the ability to place 3D objects into the same scene you're sitting in.
Picture this: you're watching an NBA Lakers game in immersive video, sitting side by side with friends. An advertiser selling shampoo could drop a 3D model of the bottle into the scene during the ad break, and everyone watching could play with it. That's useful for advertising, but it's just as useful for the game itself. Think stat overlays rendered as 3D models in real time, or the trajectory of the ball tracked and visualized on the court while the game is live. That's what this SDK enables.
What I learned about SharePlay
SharePlay is really just a shared message channel. Devices in the same group activity post messages to it. Your device isn't being directly controlled by anyone else. Person A pushes a message to the channel, something like "update UI: switch to this tab." Person B detects that message and decides whether to update their own UI. Each participant chooses, on their own device, whether and how to act on what comes through.
I figured this out using the SharePlayMock repository built by Pixeland-Tech. It let me test the whole flow on my own laptop with two simulator instances talking over a Java server running locally, so I could check whether a tab click or page navigation on one device correctly reached the other. SharePlayMock doesn't let you touch the real SharePlay API, though, so to test the actual end-to-end flow I had to ask friends I'd found through Vision Pro communities to test it with me on real devices. I've even tested it with live immersive video playback (HLS live streaming) with the help of SpatialGen's live-streaming pipeline. Thanks, Zach and Michael!
There were plenty of frustrating bugs along the way, but the process was eye-opening.
Try it yourself
The SDK is public now: github.com/juyoungk23/ImmersiveWatchParty-SDK. Fork it, open an issue, send a pull request, whatever's useful to you. I'd love to see what people build with this.
Sincerely, Juyoung
P.S. If you'd like to read another article I wrote on this in the past, check it out here.