In the Media & Entertainment world, data files have reached very large sizes, particularly in cases of high resolution video that can exceed the one terabyte in size. In order to efficiently work with very large files, the media file processing is done in sections, with the end-user requesting content “snippets” based on timecodes. Object storage devices that are used to store very large files are typically not aware of the timecode-to-byte relationship, and have no content awareness that’s necessary to extract and create partial media files. In order to bridge the gap between time and bytes, BlackPearl has added a Partial File Restore (PFR) feature to enable the media processing application to efficiently retrieve a complete media file based on timecode offsets.
In a typical Media Asset Management (MAM)/BlackPearl environment, the architecture would involve the following elements:
-- MAM – provides end user interface
-- BlackPearl MAM Plugin – handles all interactions between BlackPearl and MAM
-- BlackPearl – object storage
-- PFR Server – provides PFR processing services
-- NAS – network storage – permanent storage location for index files, temporary storage location for file processing
In the BlackPearl environment, there are two parts to the PFR processing of media files. The first part involves creating the media-aware index file for timecode/byte-offset information during file storage (writing) process. The second part of the process involves recalling a range of bytes that are based on a specified timecode, and creating complete, a self-contained media file that contains only the requested frames. In order to make this integration efficient, the PFR Server has a defined set of services that can be called by the media application using a REST interface (in this architecture – via the BlackPearl plugin).
The interface supports three basic request calls and two status calls to monitor the progress of the requested execution:
-- Request File Indexing
-- Request File Indexing Status
-- Request File Byte Offsets
-- Request Partial File Creation
-- Request Partial File Creation Status
For storing the file, the typical flow of execution would be:
- Copy media file to NAS storage
- Request creation of media index file
- Monitor status for completion
- Write file to BlackPearl Storage
For retrieving the partial file, the flow of execution would be:
- Request File Byte Offsets based on timecode
- Read specified byte range from BlackPearl to NAS
- Request partial file creation
- Monitor file creation status
- Supply partial file to Media Application
Spectra’s development team has created an C# and Java SDKs to support the rapid development and integration of the PFR with BlackPearl Storage into existing M&E environments. The SDKs provide “wrapper” calls for the REST API calls. The C# and Java client documentation and code samples can be found at:
https://github.com/SpectraLogic/tpfr_client
https://github.com/SpectraLogic/tpfr_java_client
PFR Users Guide provides the list of supported video file formats.
Quick Start Guide is a required reading to understand the services and locations that have to be setup in order to start PFR development.
PFR REST API definition is available for direct HTML development.
If you are considering creating a PFR integration with a BlackPearl environment, please contact the Spectra Logic Developer Program Team.