Video Filters in an AR Environment

a-ha's music video for their song "Take On Me" features a visually stunning effect: In this video, we witness a seamless transition between the real world and a pencil-drawn realm through the use of portals. Characters interact with these portals to enter and exit the hand-drawn world. In the past, creating such effects required extensive and costly manual labor, with each frame being manually rotoscoped. With the proliferation of AR technologies, I developed an iOS AR app that mimicked the effect found in the "Take On Me" video. In this app, you could place portals, step into the realm of the pencil drawing, explore, and interact with the portal similar to the original video.

I built the app for iOS using Apple ARKit and SceneKit. At the time of the project, the more modern RealityKit framework did not support the complex rendering manipulations required. The first step in the rendering pipeline was to stylize the video footage using Apple's CoreImage filters to give it a pencil drawing aesthetic (or any other filter supported by CoreImage). Next came the implementation of the portals themselves: I used SceneKit to create virtual green screens where the stylized video feeds would be placed. I then used custom metal shaders with SCNTechnique to render the video within these green screens. Finally, using SceneKit's capabilities, I implemented a system that allowed users to place a portal, enter it, look back, peek, and walk around it.

There are still some possibilities to explore for future iterations. One possible enhancement would be to support multiple portals that interact with each other. This would introduce a new layer of complexity, possibly requiring the use of a depth buffer to cancel out conflicting interactions between portals. Additionally, instead of manually placing the portals, the application could use the real-world geometry detected by the AR framework and automatically position the portals accordingly. This would not only streamline the user experience, but also increase the realism of the effect. One intriguing effect from the original video is the hand reaching out from the portal. By leveraging Apple's people matte capabilities, it might be possible to detect when a person is inside the portal and have their stylized body parts reach out to bridge the virtual and real worlds.