Unreal Engine 3 Powers Critical and Commercial Success LazyTown

Written by John Gaudiosi
Unreal Technology 2008

Epic Games’ Unreal Engine 3 (UE3) has already taken hold of the videogame industry, powering top games from game publishers like Electronic Arts, Sega, Midway Games, Capcom, Activision, and Sony Online Entertainment. Now, Hollywood can take notice, as the children's hit TV series, "LazyTown," is the first show utilizing the same powerful game technology to breathe life into its blend of live action, puppetry and computer-generated content. Produced and occasionally directed by Dutch television and film maker Raymond P. Le Gué, "LazyTown" currently airs around the world, including on Nickelodeon and Noggin in the U.S.


For those who don't have children, the show focuses on a "slightly above average hero" named Sportacus 10 (played by Magnus Scheving), who battles against the lazy Robbie Rotten (Stefan Karl Stefansson) to keep residents like Stephanie (Julianna Rose Mauriello) and her new friends Ziggy, Trixie, Stingy and Pixel busy outside and active instead of inside on the couch watching TV and eating junk food. The show is the brainchild of Icelandic visionary athlete Magnus Scheving who envisioned a way to entertain and educate children while encouraging them to have a healthy lifestyle. Scheving stars in the show, writes the scripts and directs most of the episodes. The show, which currently airs in 103 countries, has received three Emmy nominations and won the UK’s prestigious BAFTA award in 2006 for Best International Children’s Show.

In addition to his work as a producer and director, Le Gué has spent the past couple of decades developing, manufacturing and distributing new 3D computer animation, virtual television and cinematographic production technologies. In 1985, Le Gué founded ElectroGIG, which spent the past decade creating a high-end 3D modeling, rendering and animation software product line. At SIGGRAPH in 1994, ElectroGIG introduced the very first commercially available virtual set system on the market. Between 1995 and 2006, Le Gué focused on the development and application of real-time virtual television production pipelines and real-time motion capture and animation pipelines, resulting in a solid and proven virtual production method and four generations of virtual television production systems and architectures named XRGen1 through 4.

Le Gué explains how it all works. “We start with the live actors and puppets on a physical set with a green screen behind them as a backdrop. The green screen is replaced in real time with the sets created in XRGen4 using UE3. As we move the camera and actors around the physical set, the backdrop scene also moves in real time in complete synchronization with the movements of the real camera. All of this is recorded, and the director can watch the resulting composition in real time. For the director, this has all the qualities and freedom of working in a real set at a tiny fraction of the cost and with the complexity it would take to do this with a real set.”

Using the XRGen4 technology, Le Gué has been able to use Unreal Engine 3 as the image generator for 18 of the current 53 "LazyTown" episodes that are airing in the U.S. and in 102 other countries around the globe. "UE3 is an important part of our system," said Le Gué. "It is the tool with which we create and render our virtual sets. Until now, we have been using our own dedicated rendering engines, but when we saw the performance of Epic's game engine coming closer and closer to our needs, we decided this is something we needed to investigate."

Le Gué said the studio setup allows the team to generate the "LazyTown" virtual environment, a 3D townscape, as a live background in real time and in high definition. Chromakey (green screen) techniques are used to key these backgrounds behind the live actors.

"This implies that we have a finished composited image from the floor," explained Le Gué. "The camera and crane movements are communicated through sensors to our XRGen4 software to the UE3 game engine. The concept and production design of 'LazyTown' calls for a very rich and diverse look with many location changes within the environment. We would have never been able to do this without the use of virtual sets and hence our system and the UE3 game engine."

In every 24-minute "LazyTown" episode, there are between 400 and 600 visual effects shots, separate from all the shots that are already composited live. Le Gué said producing television like this calls for more planning than a traditionally shot show. In essence, every shot is a potential visual effects shot; some are recorded live, while other shots are processed in post production. Aiming for a highly stylized result, where virtual backgrounds are combined with live foregrounds, has an impact on all design disciplines. For example, virtual sets and physical sets have to match, costumes and props have to match, and, due to the fact that everything is shot in high definition, the quality demands of all used material are extremely high.

UE3 has opened up new avenues for Le Gué and his creative team. They have been able to have free camera movement in real time and genlocked on HD resolution with an extremely large set and a lot of textures in memory.

"This was the first time we used the UE3 engine, and we were extremely pleased that we were able to make the game engine work for us in this very demanding professional production environment," said Le Gué. "For our next production, we will explore more of the extended rendering qualities of UE3 and work them into our production design."

Le Gué said it was pure coincidence that he stumbled upon UE3 for "LazyTown." He said he's always on the lookout to find new ways to create television content, especially in the fantasy/drama genre and the TV game show genre.

"My wife, who is designing game shows for TV, came in contact with Epic at a Game Developers Conference and we started to look into using Unreal Engine for game shows," said Le Gué. "Meanwhile, I was asked to work on 'LazyTown' as executive producer/director on the basis of my extensive experience with virtual sets. About a year later, one the visual effects artists at 'LazyTown' loaded parts of the virtual set into the Unreal game engine. Seeing that made us realize that we should explore the possibility to use UE3 for the second season of the show. Fortunately, we already had our own XRGen3 technology available and we only needed to swap the render core and voila--XRGen4 with UE3."

Le Gué has moved on from "LazyTown" and is now developing some original film productions that will make further use of the virtual sets with XRGen4 and UE3. He believes that UE3 technology, once Hollywood gets its hands on it, will have a watershed impact on the movie and TV business.

"This technology is not only opening doors to Hollywood but independent producers around the world," said Le Gué. "When you realize that the production technologies and methods of the movie '300' are the same as the ones that we have applied on the production of 'LazyTown' and that the underlying hardware platforms are basically the same as the ones that are being used by videogame makers, then the equation is very simple. The impact will be felt everywhere and it will be another driving force, next to high definition, to further convergence of film, TV and games."

Le Gué compares "300" and its gaming influence to movies like "Jurassic Park" and "Terminator 2," which were key touchstones in the visual effects movement. Thanks in part to UE3, Le Gué said Hollywood and the creative forces around the globe that utilize this technology are on the cusp of this type of wave.