SMPTE Newswatch Masthead

Hot Button Discussion

Augmented Reality Light Fields
By Michael Goldman  
 
Of the many interactive digital technology platforms designed to offer modern consumers new entertainment experiences, none have the paradigm-changing potential of so-called virtual reality and augmented reality technology. This was clearly demonstrated earlier this year when Facebook announced a massive $2.3-billion-dollar acquisition of videogame start-up company Oculus VR, the company that created the Oculus Rift virtual reality headset. The acquisition was part of a Facebook strategy to not only move into the space generally, but also to explore whether virtual reality (VR) technology that had originally been developed for professional-oriented, simulation applications in the military, medical, and industrial design worlds, and later ported over to the gaming universe, might eventually evolve into a platform to build the ultimate, interactive, consumer shopping and entertainment playground of the future.
 
Still, as companies like Facebook, Google, Microsoft, Sony, and others explore such possibilities, for the time being, in the entertainment world, gaming remains the only place where interactive VR applications have prospered. In fact, great confusion reigns about  proper role or impact of such technology in the long-term for the cinema and broadcast content creation and distribution industries. It can be a perplexing landscape because, among other things, it incorporates questions about how various new broadcast display and distribution advances, such as higher resolution displays, 3D, immersive audio, higher frame rates, greater bandwidth, faster rendering, touchscreen and other interactive technologies might be incorporated and combined into new virtual reality entertainment platforms.
 
Certainly, content creators and distributors are interested in the new potential  revenue streams that VR technology might introduce. Determining how to accomplish this, and whether it is even feasible, affordable, and worthwhile for the entertainment industry, are issues that loom over every aspect of the discussion, suggests industry consultant Dave Singhal, who advises companies on how to exploit new interactive display technologies in the light-field industry. Singhal participated in the SMPTE Entertainment Technology in the Internet Age (ETIA) conference at Stanford University this past June and helped organize  a special evening session called "The Holodeck: Entertainment for the Next Generation." During that session, inspired by the ultimate sci-fi realization of the VR concept from the "Star Trek" television franchise, called the Holodeck, attendees were able to view and interact with a series of experimental new technologies from a variety of companies and research projects. The purpose was to give an idea of where research into augmenting reality for entertainment purposes beyond traditional gaming applications might be going.
 
Star Trek's Holodeck might be the end fantasy for those who believe in such technology--a concept in which an entirely artificial, computer-generated environment can be created for users to completely interact with physically. Singhal says there is actual research into such concepts, but for now, the more realistic notion of "augmented" reality, in which computer-generated sensory inputs are added in various ways to goggles, helmets, gloves, screens, or an entire immersive room, are not only possible, but well under way for different types of applications. For example, he points to a wide range of products, demos, and presentations that were on display this past June at the industry tradeshow dedicated exclusively to the world of augmented reality--Augmented World Expo 2014.
 
One problem, he suggests, involves getting entertainment companies to wrap their minds around how to define virtual reality for their own purposes, and understand what the proper niches of such technologies might be for consumer entertainment beyond the videogame space.

"Imagine an iPad without the touch capability, it would not be much better than a TV or picture frame," Singhal says. "The same thing applies with what we call holographic images or light-field imaging, whether you experience them head-mounted or on a display on the wall. You need some way to make them interactive. If your display shows a vase of flowers, you want to be able to pick up the vase and move it, with the scene looking real, and your hand only moving the object when your fingers encircle it. That is what we are striving for--that is when it truly becomes a compelling experience. In the last year or two, we have had camera-based controllers like Leap Motion and Kinect come along, and they allow for strongly-registered and very accurate measurements of where your hands are, and therefore, with that kind of technology, we can start to combine that with the holographic content that is coming out to the user. So for right now, what is attainable is to make the display holographic and the control sensors volumetric, so that objects behave naturally and interactively with your fingers in combination with physical modeling software. Right now, those things are essentially here, but face cost and form factor obstacles. For instance, the displays are expensive if you mount them on a wall, but are a lot cheaper if you wear them on your head [via goggles or a helmet], yet a head-mounted display is not a great form factor."
 
And that is precisely the problem, Singhal continues. Consumers seeking entertainment beyond the hardcore gaming crowd do not necessarily want to have a bunch of gear in their own homes in order to be entertained. Indeed, 3D television technology has not become the big success in the United States that many predicted for precisely this reason, and that [3DTV] requires only a pair of polarized 3D glasses. Therefore, for the time being, he suggests, the debate is about "what can make a system compelling and attainable enough for consumers to want to try it out and even use it past the initial interest."
 
Obviously, keeping the cost down is one thing that can make a system "attainable." Therefore, "true" virtual reality in the sense of fully interactive holographic displays in consumer homes is not so much what developers are pursuing. While wall-sized interactive displays are technically possible today, the cost barrier associated with them has limited interest from potential content developers, Singhal suggests. In addition, he points out that consumers and the entire market tend to get confused by the term "holographic," which has been used rather loosely across the industry in recent years. For the systems available now or currently in development, the central component does not so much revolve around generating true holographic images, but rather, "what can more accurately be described as non-Lambertian or 3D light fields," he states.  
 
Singhal explains that there are three types of light fields. The first is a 3D light field that, in the best case, is like the real world, whereby from every point in space where your eyeball might be, you see something just from that point in space. You can position your eye in three dimensions, and point it in three different directions, and expect to see something different for each position and orientation. That is the best-case possible light field.
 
"The next case is to get rid of the vertical component, but retain horizontal parallax. You will only see things horizontally different. For example, a 3DTV display, like some now being offered with two or more views, in which you can move your head horizontally and see a limited number of different viewing angles into that scene. This is called horizontal-only parallax light field. When you remove the vertical information, it becomes much easier to handle the data and display," Singhal says.
 
The last case, Singhal explains, "is a regular TV on a wall, where there is a Lambertian filter in front of every pixel, meaning that the light coming out of each pixel is going out with the same color in all directions--sort of a hemispherical direction, but with the same color. You move around and look from different angles, but that pixel is the same color, so nothing changes--traditional 2D, or you could call it a 2D light field."  
 
"You can get any of these with a head-mounted display, which is interesting because you can move your head in any direction and, with a well-developed tracking system, each eye will get the appropriate light that should be entering your eyeball for where the image is in space and orientation. Or you can get that with a 2D display that tracks where you are. As you move, it changes what it is showing you--a kind of tracking system to give you a light-field effect."
 
Such "effects" are readily available with a wide range of new applications that are relatively inexpensive to offer to consumers. Lately, there has been a new VR trend beyond the high-end gaming systems that Oculus and others represent. That trend is in the opposite direction--toward cheap apps and goggles that are simply fun and easy to cobble together.
 
The most popular example of this approach has been the Google Cardboard project, which is, in essence, as Google promotes it, "virtual reality on your Smartphone." Instead of specialized hardware, Google asked developers to determine ways to transform smartphones into VR headsets running mobile apps. The Google Cardboard project involves a way to build a pair of cardboard goggles with basic lenses to hold your phone while running a special app, and strapping the "goggles" to your head, so that, while maneuvering your head back-and-forth, users can enjoy a VR-style experience. (Here is a  YouTube video on how this works from CNET.)

Ironically, Singhal points out, at the SMPTE ETIA conference in June, just days before Google unveiled the Cardboard project to developers at its I/O Developers Conference, a group of high school students from Cupertino, California, demonstrated their own version of an almost identical concept called Face Box.  
 
"These things show us that there is value in doing virtual reality the cheap way for entertainment value," Singhal states. "It's very expensive to make a horizontal-only parallax display. It is ultra expensive to make a horizontal and vertical parallax holographic display. But as these projects show, it is incredibly cheap to track the position of your eyes by putting the display up near your face with a couple of lenses in front of each eye, and then send the appropriate game or software-generated content to each eyeball. That kind of tracking system is happening right now. They are working hard to make it attainable and to popularize the platform. But the unanswered question is, if only about 10% of the people wanted 3D televisions, is that the same forecast for head-mounted displays, no matter how affordable? Will people want to stick something on their head all the time? So the question of adoption is really the big question."
 
For the entertainment industry, the connected question then moves into the world of content creation, and how to fold VR possibilities into traditional content like motion pictures or television shows. Obviously, Hollywood-connected companies are now exploring this world, even if it is still far from clear how they can or will connect it to more traditional forms of media. Singhal says he wonders if we will see interactive bonus content on Blue-ray discs or pay-per-view material for those interested in traveling this road, and how exactly the industry "might reinvent content," to accommodate what is possible with augmented reality presentations. Some facilities around the industry have started ramping up production to create VR-related content for clients as part of this embryonic reinvention. In September, for example, popular visual effects' vendor Reel FX announced the creation of a dedicated VR division, and others are expected to follow.
 
"There is certainly a lot of activity out there," Singhal says. "There are companies like Jaunt that are trying to offer cinematic, 360-degree recording solutions with the hope that people will use them to possibly record sporting events, like a football game, and allow viewers to put themselves into the huddle or onto the sideline, by moving their heads. There is a lot of exploration in that regard, but as they reinvent content, it is far from clear how mainstream consumers will respond, he says. "I don't think they have found broadly applicable content for it yet, beyond games and simulators, like first-person shooter type of immersive games and defense training systems, but there are a number of vertical entertainment applications that are really very cool."
 
There is also the related issue of how to efficiently mix these new kinds of content into the mastering pipeline in a multiplatform world, packaging it together with all the other kinds of data that are now part of the equation. "Content creators, by definition, would have to shoot more angles, cut more scenes together, package that material together in different ways, and then add it all to an already crowded mastering process for the 'VR version' of some movies or TV shows potentially," Singhal adds.

"The whole point of immersive light field content is to see the scene from a variety of viewing angles," he says. "Today, that is not the typical way they would make a movie, and the concept conflicts with the creative idea of shooting the program from the director's point of view, which would include a narrowly- defined range of angles; that would require a cultural shift. For now, would they want to do this just for people either wearing head-mounted displays or investing in expensive wall-mounted light field displays?"
 
"Also, the quality requirements are different typically for a motion picture than for a videogame, for example. You would almost have to invent a new quality metric for content in this regard. For display technology, the main quality metrics are resolution, interactivity, and the speed of the scene. For virtual reality, the more you get one or the other, the less you might need of another for a particular scene. The faster the scene, the less you would need to worry about resolution. If you have a lot of interactivity, you can drop down the resolution. Less interactivity would [require] more resolution to make it acceptable. These are all new ways of thinking about the material that content creators would have to consider." 
 

News Briefs

Netflix Movie Moves
Hollywood trade publications were all abuzz, as October dawned, about recent moves by streaming content giant Netflix to break, in a manner of speaking, into the movie business. The recent dual announcements in the movie space involved a deal with the Weinstein Company to produce a sequel to Ang Lee's 2000 film, Crouching Tiger, Hidden Dragon, for simultaneous release into selected IMAX theaters worldwide and onto Netflix's streaming video service next summer. Then, a few days later, Netflix announced it had signed movie star Adam Sandler to an exclusive four-picture deal for movies that will be produced with Netflix and then premiere exclusively to Netflix members in dozens of countries across the globe before hitting movie theaters. In a recent column, Variety co-editor Andrew Wallenstein suggested this was part of a move to help Netflix lead a challenge to the traditional ways that motion picture distribution works, much as the streaming video service had previously challenged how traditional broadcast and cable television models worked in recent years, particularly with the debut of its first original dramatic program, "House of Cards" in 2011. Wallenstein suggests that such moves may not be about trying to compete with motion picture exhibition, but rather about broadening usage of gaming consoles, Blu-ray players, and various other streaming media devices and smart TVs to bring high-end streamed content into the home. He adds that budget limitations and limits on theatrical and studio partner cooperation mean such a paradigm shift remains far from posing a major threat to the traditional motion picture exhibition methodology. But it does, he says, illustrate the growing ability of broadband-powered streaming media services to "skim cream off the top" of some traditional entertainment business models, as Netflix has already illustrated.

Drones OK for Hollywood
The Federal Aviation Administration agreed at the end of September to grant operator exemptions to six aerial production companies in the U.S., permitting them to use small unmanned aircraft systems (UAS)--drones--for filming movies and TV shows. As discussed in the August Newswatch, filmmakers have increasingly been finding highly-creative ways to use drone-based cinematography techniques, but had consistently been encountering roadblocks where regulatory limitations on unmanned aerial vehicles were concerned. U.S. Transportation Department officials announced at a press conference in late September that they had decided that such applications did not pose any threat to national or local security, and so they granted the exemptions at the request of the six production companies and the MPAA. That same day, SMPTE hosted a Hollywood event to discuss the creative potential of using drones in different kinds of cinematography applications. Read the Hollywood Reporter's summary of that event here.