Case Study: Creating a Digital Innovation Culture

Digital Business

Digitization of organizations, the mapping of digital models onto physical objects and business processes (or business systems), is important in learning how organizations actually work, and learning how to reduce costs by improving the business system. The ultimate improvement is to automate business processes. For this to be possible, it is necessary to incrementally improve the digital models until there is a very high correlation between actual and modeled behavior. What starts off as simple digital models become high complex, and utilize huge amounts of processing and data resources.

In thinking about this as a business process, it is clear that film companies are in the vanguard of this innovation.   Wikibon spoke to Weta Digital about their digitization process, to Dave Gouge, VP of Publicity and Marketing at Weta Digital, and Matt Cunningham, Head of Systems.

This research is geared to understand better what the key business and IT processes are required for success. And, of course, to look for an answer to the burning question of whether actors would follow professional drivers as future casualties of artificial intelligence.

Creating an Digital Innovation Culture at Weta

Visual effects (abbreviated VFX) are the processes to create and manipulate film imagery without taking traditional live action shots. Weta Digital is a VFX company founded by Peter Jackson and others in 1993 to support his films made in New Zealand, where no VFX companies existed. The film adaptation of Tolkeins “Lord of the Rings” proved to be a strong catalyst for growth, from 300 people in 2001 to 1,400 now. Weta Digital now supports many other film studios in addition to Peter Jackson’s. To date, Weta Digital has won five Academy Awards for Best Visual Effects. The latest was in 2009, when Joe Letteri won for Avatar.

Weta Digital believe that the keys to their business success are attracting the best artists in the world and creating a creative-led company. Joe Letteri, the Senior Visual Effects Supervisor, in a key innovator who continues to drive change within the company. The key creative imperatives for true visual effects innovation are the ability of the artists to try out ideas and fail fast, and then iterate successful ideas forward as fast as possible to completion. Rapid feedback from all in the creative process team has created the fundamental culture with Weta Digital. This top-down director-led culture allows confidence that Weta Digital can deliver innovation, deliver on time, and can reinvent the potential for movie revenue generation.

Anybody who has seen their academy best visual effects award winning films (The Lord of the Rings: The Fellowship of the Ring in 2001, The Lord of the Rings: The Two Towers in 2002, The Lord of the Rings: The Return of the King in 2003, King Kong in 2005, and Avatar in 2009) knows the amazing creativity that has emanated from Weta Digital. Figure 1 shows an example.

IT Imperatives for a Digital Business

Behind this creative process is an IT department who has constantly had to push the boundaries of compute, networking, storage and software to be an enabler of this creative process and not a bottleneck.

The first IT imperative is that to lead in innovation, the software has often to be created from scratch. Creating giant digital waterfalls is a big thing; or you can wait for packages to catch up to give you passable digital water. Understanding the business impact on film revenues is essential to deciding whether awe-inspiring or passable is the best investment. There has to be understanding and trust between the Director, the creative staff and IT.

Digital Innovation Focus

Weta Digital have tended to take a slightly longer view on what challenges they face in the future. They have invested heavily in fire, water and fur systems, and are fundamentally known as a creature shop. Weta have spent time simulating in immense detail how muscles actually fire and the surrounding effects on the flesh and fur, things on the capability edge. Weta are well known for their motion capture technology, and the development of really smart software algorithms. The have an excellent framework to build out motion and vision tracking and providing support to animators to aid their processes.

Weta is driving innovation to enable things that people haven’t seen before and are using that approach to drive innovation and efficiency. Excellence is enabled by the ability to iterate and keep trying again and again and continuing to iterate until they meet the directors needs. A fundamental systems requirement for the artists is quick turnaround.

This puts a large stress on the IT systems that enable those artists to work at the speed that is required by directors. The speed of turnaround for these films and the growth in the numbers of films (the growth of the Marvel films alone has put a ton of visual effects shots in the marketplace) emphasizes the imperative for IT not to be a impediment to creating business value for Weta Digital.

IT supporting Digital Weta

Weta Digital needs very good response times for all data requirements, to improve the productivity of the creative artists. Cloud systems could not support the response times required; the fact that Weta was based in New Zealand only added to the latency challenges! Weta chose onsite support.

Render Wall

The key delivery vehicle for delivering the VFX results to the team is the render wall, six rows  rows deep. A wall of digital displays is used by the directors and the creative teams to give feedback on progress, to eliminate ideas that do not work, and to iterate on ideas until completion. Supporting the render wall are hundreds of screens for individuals working on effects before and after render wall evaluation.

Processing Hardware

The processing of the software is done a racks of HPE servers, with very large amounts of main memory to minimize the transport of data to the processors, and keep the processors busy. Three of the six rows in the render wall have been replaced with the latest Intel Haswell CPUs.

The Network

At the center of the Weta network handling all the high throughput  is an Arista 7508, the fastest available. Storage and processors are attached to the same switch. Weta is looking at upgrading to 40Gb or 100Gb interfaces. The rest of network is mostly Juniper. Weta do not use jumbo frames, and will be evaluating this complex process  at some point.

The File System

When Weta first came to market there were no global file systems with the functionality that Weta needed to meet the business needs of the company. Weta’s global namespace, DSMS, spreads the load across all of its storage systems and migrates data from one tier to another seamlessly. DSMS was designed in-house at Weta, and enables the company to change out parts of its architecture, such as its NAS systems, and implement new systems that can seamlessly be integrated into the overall Weta VFX workload. Weta’s Global Namespace operates at a higher plane and abstracts the underlying hardware from the file system, enabling Weta to shift resources around dynamically, on-the-fly, to better match the right compute and storage resources with VFX requirements. This gives Weta the ability to alter the storage infrastructure architecture underneath to allow the company to take advantage of the latest technology or latest thinking from all of its technology partners.

The Storage System

There are now 4 petabytes of production data onsite, and an offsite archive of 16 petabytes.  The data is mostly generated by the RenderWall, is incompressible and cannot be de-duplicated. About 200 to 300 terabytes of data are transferred to and from the storage system every night, a very large amount of data.

The data is widely striped over a large number of disks, and large blocks at every part of the storage system is essential to achieve throughput. Weta Digital used BlueArc filers a few years ago, and still use NetApp filers with Avere as front-end caches. These systems could not economically keep up with the ever increasing data rates required to support the RenderWall.

The current storage environment now has an additional six Oracle ZFS Appliances. Five are onsite, and 2 travel with the film units. The key advantage of the Oracle ZFS Appliances over the existing systems was a 2x improvement in bandwidth, and the elimination of the need for the Avere caching layer, which makes the overall system much more cost effective.

According to Matt Cunningham, Weta are getting “twice the performance and capacity for half the cost, support costs and ongoing maintenance. We could light up the RenderWall and the ZFS systems will fill it up”. In one specific read-heavy application, Weta have achieved 36 terabytes/hour throughput and 400,000 IOPs from a single ZFS appliance, and is planning to install more 10Gb ports to increase the throughput further.

Geek Detail on Storage System Design

To achieve this high bandwidth, the fundamental L1 cache block size in the controller has to be as high as possible. For very large sequential files, the L2 flash cache is not useful. The ZFS uses an Adaptive Replacement Cache (ARC) mechanism, and Oracle tuned the ZFS to use a 1 MB block size. The previous filers (NetApp) are constrained to 64K. The use of the  inside the very large memory and tuning the ARC in the ZFS controllers is instrumental in achieving the 80 terabyte/hour throughput. Further detail from Wikibon on the ZFS architecture is “Oracle ZFS Hybrid Storage Appliance Reads for Show but Writes for Dough”

Will Actors be Needed at All?

Dave Gouge says that; “All CG characters is not far away and is infinitely far away”. His view is that it will not be the norm and it will be very unlikely to see it any time soon.The reason is that it is a very difficult process, taking an enormous amount of time and effort. It is much easier to collaborate with an actor, to get the creative inputs from both sides.

When it is just too dangerous for stunt people perform the stunts, a digital double, an digital actor is created. They are used in many places, enabling interaction between the human actor and the digital character. In the third installment of the Hobbit trilogy, all of those battle sequences, all of the elves and Orcs etc., are entirely VFX characters. This capability add enormous value to the creativity and impact of the film. But full-on digital characters without any humans in an entire film is currently too much work and time. Even if it was possible, visual artists would have to become actors themselves. It is easier and create better quality to use motion capture technology together with a real live actor.

Conclusions on Creating a Digital Business

The key aspects of creating a Digital Business seems to be a very close connection between the professionals creating the digitization and the executives creating business value. The ability to create a working environment that can encourage iteration, be able to fail fast and integrate again, and be able to judge the business value of the digital models created. IT needs to focus on not getting in the way, but providing the technology needed to ensure fast iteration, and being comfortable with writing software from scratch to solve problems.

Weta Digital could create something more than the ability to win academy awards. They could be in the business of helping other organizations create a viable digital business. There are no enterprise organizations that have married the business, digital creativity and IT as seamlessly as Weta Digital.