Netflix and the IMF Community

Photon OSS - IMF validation for the masses

When you’ve got something this good, it’s hard to keep it to yourself. As we develop our IMF validation tools internally, we are merging them into a public git repository. The project, code-named Photon (to imply a torchbearer), was envisioned to aid a wider adoption of the IMF standard, and to simplify the development of IMF tools. It could be utilized in a few different ways: as a core library for building a complete end-to-end IMF content ingestion workflow, a web-service backend providing quick turnaround in validating IMF assets or even as a reference implementation of the IMF standard.

Photon in the Netflix Content Processing Pipeline

IMF presents new opportunities and significant challenges to the Digital Supply Chain ecosystem. Photon embodies all our knowledge and experience of building an automated, distributed cloud-based content ingestion workflow. At the time of this blog, we have fully integrated Photon into the Netflix IMF workflow and continue to enhance it as our workflow requirements evolve. A simple 3-step diagram of the Netflix content processing workflow (as discussed in our last tech blog Netflix IMF Workflow) along with the usage of Photon is shown in the figure below.

Photon has all the necessary logic for parsing, reading and validating IMF assets including AssetMap, Packing List (PKL), Composition Playlist (CPL) and Audio/Video track files. Some of the salient features of Photon that we have leveraged in building our IMF content ingestion workflow are:

1. A modular architecture along with a set of thread safe classes for validating IMF assets such as Composition Playlist, Packing List and AssetMap.
2. A model to enforce IMF constraints on structural metadata of track files and Composition Playlists.
3. Support for multiple namespaces for Composition Playlist, Packing List and AssetMap in order to remain compliant with the newer schemas published by SMPTE.
4. A parser/reader to interpret metadata within IMF track files and serialize it as a SMPTE st2067-3 (the Composition Playlist specification) compliant XML document.
5. Implementation of deep inspection of IMF assets including algorithms for Composition Playlist conformance and associativity (more on this aspect below).
6. A stateless interface for IMF validation that could be used as a backend in a RESTful web service for validating IMF packages.

IMF Composition in the Real World

As IMF evolves, so will the tools used to produce IMF packages. While the standard and tools mature we expect to receive assets in the interim that haven’t caught up with the latest specifications. In order to minimize such malformed assets from making their way into our workflow we are striving to implement algorithms for performing deep inspections. In the earlier section we introduced two algorithms implemented in Photon for deep inspections, namely Composition Playlist conformance and Composition Playlist associativity. We will attempt to define and elaborate on these algorithms in this section.

Composition Playlist Conformance

We define a Composition Playlist to be conformant if the entire file descriptor metadata structure (including sub-descriptors) present in each track file that is a part of the composition is mapped to a single essence descriptor element in the Composition Playlist’s Essence Descriptor List. The algorithm to determine Composition Playlist conformance comprises the following steps:

1. Parse and read all the essence descriptors along with their associated sub-descriptors from every track file that is a part of the composition.
2. Parse and read all the essence descriptor elements along with their associated sub-descriptors in the Essence Descriptor List.
3. Verify that every Essence Descriptor in the Essence Descriptor List is referenced by at least one track file in the composition. If not the Composition Playlist is not conformant.
4. Identify the essence descriptor in the Essence Descriptor List corresponding to the next track file. This is done by utilizing syntactical elements defined in the Composition Playlist - namely Track File ID and SourceEncoding element. If not present the Composition Playlist is not conformant.
5. Compare the identified essence descriptor and its sub-descriptors present in the Essence Descriptor List with the corresponding essence descriptor and its sub-descriptors present in the track file. At least one essence descriptor and sub-descriptors in the Track file should match with the corresponding essence descriptor and sub-descriptors in the Essence Descriptor List. If not, the Composition Playlist is not conformant.

The algorithmic approach we have adopted to perform this check is depicted in the following flow chart:

Composition Associativity

The current definition of IMF allows for version management between one IMF publisher and one IMF consumer. In the real world, multiple parties (content partners, fulfillment partners, etc.) often work together to produce a finished title. This suggests the need for a multi-party version management system (along the lines of software version control systems). While the IMF standard does not preclude this - this aspect is missing in existing IMF implementations and does not have industry mind-share as of yet. We have come up with the concept of Composition associativity as a solution to identify and associate Composition Playlists of the same presentation that were not constructed incrementally. Such scenarios could occur when multiple Composition Playlist assets are received for a certain title where each asset fulfills certain supplemental tracks of the original video presentation.  As an example,  let us say a content partner authors a Composition Playlist for a certain title with an original video track and an English audio track, whereas the fulfillment partner publishes a Composition Playlist with the same original video track and a Spanish audio track.

The current version of the algorithm for verifying composition associativity comprises the following checks:

1. Verify that the constituent edit units of the original video track across all of the Composition Playlists are temporally aligned and represent the same video material. If not, the Composition Playlists are not associative.
2. Verify that the constituent edit units of a particular audio language track, if present, in multiple Composition Playlists to be associated are temporally aligned and represent the same audio material. If not, the Composition Playlists are not associative.
3. Repeat step 2 for the intersection set of all the audio language tracks in each of the Composition Playlist files.

Note that as of this writing, Photon does not yet have support for IMF virtual marker track as well as data tracks such as timed text, hence we do not yet include those track types in our associativity checks. A flow chart of the composition associativity algorithm follows:

How to get Photon, use it and contribute to it

Photon is hosted on the Netflix GitHub page and is licensed under the Apache License Version 2.0 terms making it very easy to use. It can be built using a gradle environment and is accompanied by a completely automated build and continuous integration system called Travis. All releases of Photon are published to Maven Central as a java archive and users can include it in their projects as a dependency using the relevant syntax for their build environment. The code base is structured in the form of packages that are intuitively named to represent the functionality that they embody. We recommend reviewing the classes in the “app” package to start with as they exercise almost all of the core implementation of the library and therefore offer valuable insight into the software structure which can be very useful to anyone that would like to get involved in the project or simply wants to understand the implementation. A complete set of Javadocs and necessary readme files are also maintained at the GitHub location and can be consulted for API reference and general information about the project.

In addition to the initiative of driving adoption of IMF in the industry, the intention for open sourcing Photon has also been to encourage and seek contributions from the Open Source community to help improve what we have built. Photon is still in its early stage of development making it very attractive for new contributions. Some of the areas in which we are seeking feedback as well as contributions but not limited to are as follows:

1. Software design and architectural improvements.
2. Well designed APIs and accompanying Java documents.
3. Code quality and robustness improvements.
4. More extensive tests and code coverage.

We have simplified the process of contributing to Photon by allowing contributors to submit pull requests for review and/or forking the repository and enhancing it as necessary. Every commit is gated by a test suite, FindBugs and PMD checks before becoming ready to be accepted for merge into the mainline.

It is our belief that significant breakthroughs with Photon can only be achieved by a high level of participation and collaboration within the Open Source community. Hence we require that all Photon submissions adhere to the Apache 2.0 license terms and conditions.

Netflix IMF Roadmap

As we continue to contribute to Photon to bridge the gap between its current feature set and the IMF standard, our  strategic initiatives and plans around IMF include the following:

1. We are participating in various standardization activities related to IMF. These include our support for standardization of TTML2 in W3C as well as HDR video and immersive audio in the SMPTE IMF group. We are very enthusiastic about ACES for IMF. Netflix has sponsored multiple IMF meetups and interop plugfests.   
2. We are actively investing in open source software (OSS) activities around IMF and hoping to foster a collaborative and vibrant developer community. Examples of other OSS projects sponsored by Netflix include “imf-validation-tool” and “regxmllib”.
3. We are engaged in the development of tools that make it easy to use as well as embed IMF in existing workflows. An example of an ongoing project is the ability to transcode from IMF to DPP (Digital Production Partnership - an initiative formed jointly by public service broadcasters in the UK to help producers and broadcasters maximize the potential benefits of digital television production) or IMF to iTunes ecosystems. Another example is the “IMF CPL Editor” project. This will be available on GitHub and will support lightweight editing of the CPL such as metadata and timeline changes.
4. IMF at its heart is an asset distribution and archival standard. Effective industry-wide automation in the digital supply chain could be achieved by integration with content identification systems such as EIDR and MovieLabs Media Manifest and Avails protocols. Netflix is actively involved in these initiatives.
5. We are committed to building and maintaining scalable web-services for validating and perhaps even authoring IMF packages. We believe that this would be a significant step towards addressing the needs of the IMF community at large, and further helping drive IMF adoption in the industry.

The Future of Photon

Photon will evolve over time as we continue to build our IMF content ingestion workflow. As indicated in the very first article in this series “IMF: A Prescription for Versionitis” Netflix realizes that IMF will provide a scalable solution to some of the most common challenges in the Digital Supply Chain ecosystem. However, some of the aspects that bring in those efficiencies such as a model for interactive content, integration with content identification systems, scalable web services for IMF validation, etc. are under development. Good authoring tools will drive adoption and we believe they are critical for the success of IMF. By participating in various standardization activities around IMF we have the opportunity to constantly review possible areas of future development that would not only enhance the usability of Photon but also aid in IMF adoption through easy-to-use open source tools.

Conclusions

IMF is an evolving standard - while it addresses many problems in the Digital Supply Chain ecosystem, challenges abound and there is opportunity for further work. The success of IMF will depend upon participation by content partners, fulfillment partners as well as content retailers. At this time, Netflix is 100% committed to the success of IMF.

By Sreeram Chakrovorthy, Rohit Puri, and Andy Schuler