Composite View Recording and Screenshot Capture in Apple Vision Pro

[Carton22]

Hi @from2001,

Fantastic repo and meaningful tool for XR developers!
I wonder if this plugin will be able to capture the composite view inside Apple Vision Pro?
Based on my understanding, the current Main Camera Access will only capture the physical world view without a digital element.
But since I want to work on AR, I need to capture both the physical real-world environment and all the digital virtual elements overlaid on it.
For example, if you use system-level screen recoding, you will get the composite views.
Or if you know any programmatic way in Unity or Xcode we will be able to send a command, and trigger the start of composite view screenshot capture?

Look forward to your response!

[from2001]

Hi @Carton22

Apple has added a new API called “Passthrough in screen capture – Access a composite feed of what an Apple Vision Pro wearer is seeing (physical world and digital content).”

From the documentation, it looks like this API should allow exactly what you’re asking for: capturing the composite view that includes both the real-world passthrough and the overlaid virtual content.

However, this functionality requires a specific entitlement. I don’t have that entitlement yet, so I’ve submitted a request to Apple to get it added to my account.

Once Apple grants the entitlement, I’ll test the API.

[Carton22]

Thanks @from2001 !

Yes, I know the entitlement “Passthrough in screen capture – Access a composite feed of what an Apple Vision Pro wearer is seeing (physical world and digital content).”. It is one of the entitlements you will get if you have Enterprise API for Apple Vision Pro development.
But I am not sure from the code, how we use this entitlement.
Based on my research, maybe ReplayKit is something we need.
But I have never found a sample code for VisionOS, and not sure how to do it to actually get the screenshot or screen recording of the view.
Thank you for checking that out!

[Carton22]

I currently found a workaround to fulfill my need and be able to get the composite view.
AppleVisionPro_CompositeView_Capture
It does not require Enterprise API / Xcode / SwiftUI, I recommend checking about it and hope it helps!

[from2001]

Actually, this plugin is not purely written in C# code. I made a bridge swift code to transfer frame data to Unity. I will need to write similar code to make the composite video capture feature in Unity.

I'll take a look, anyway.

[Carton22]

sounds good! Hope to get the composite video capture feature in your plugin soon!
Thanks for the hard work!