The idea of being able to explore the 3D environment of a game while still monitoring the progress of a single person is far from a new idea, but research is starting to trend toward making that pipe-dream a reality.

Streaming services like Twitch take a single player's rendered view of a game and broadcast it out to thousands of viewers as a 2D video stream, an approach that helps further popularize video games as a spectator sport. The ability for users to change the perspective of the live stream they're viewing, however, would make the experience even more compelling (much like alternate camera angles in a sports match). 

Researchers from the University of Waterloo's Cheriton School of Computer Science in Ontario, Canada, have come up with a way to give viewers of a video game stream the ability to actually look around in the 3D world being presented - without having to own a copy of the game. Even mobile devices or tablets that could never run the game would be eligible for this technology. If you'd like to read the research paper for yourself, check it out here.

Researchers enhance the 2D video data sent to viewers with additional information that's pulled from the game's real-time rendering engine, including the "depth buffer, camera pose, and projection matrix." On the viewer's end, this data allows the video information to be used to recreate a 3D environment that matches the geometries from the game. Just like in game spectator modes, the viewer can manipulate the environments to change where they're looking or even where the camera is positioned. 

The drawback of this is that the recreated environment lacks texture and graphical information, so it won't be especially pretty to look at. Still, it's a good place to start with plenty of room for improvement and development!

What do you think about this kind of experience, would you enjoy it during a livestream?