Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We are pleased to announce that now we have an SDK for the Go programming language to simplify and accelerate BlackPearl client application development. In addition to Go, we currently support SDKs for Java, C#/.NET, C and Python. You can download the Go SDK and view the Go SDK documentation. This is a new language release, so please send us any feedback you might have on this SDK.
In the Media & Entertainment world, data files have reached very large sizes, particularly in cases of high resolution video that can exceed the one terabyte in size. In order to efficiently work with very large files, the media file processing is done in sections, with the end-user requesting content “snippets” based on timecodes. Object storage devices that are used to store very large files are typically not aware of the timecode-to-byte relationship, and have no content awareness that’s necessary to extract and create partial media files. In order to bridge the gap between time and bytes, BlackPearl has added a Partial File Restore (PFR) feature to enable the media processing application to efficiently retrieve a complete media file based on timecode offsets.
In a typical Media Asset Management (MAM)/BlackPearl environment, the architecture would involve the following elements:
-- MAM – provides end user interface
-- BlackPearl MAM Plugin – handles all interactions between BlackPearl and MAM
-- BlackPearl – object storage
-- PFR Server – provides PFR processing services
-- NAS – network storage – permanent storage location for index files, temporary storage location for file processing
In the BlackPearl environment, there are two parts to the PFR processing of media files. The first part involves creating the media-aware index file for timecode/byte-offset information during file storage (writing) process. The second part of the process involves recalling a range of bytes that are based on a specified timecode, and creating complete, a self-contained media file that contains only the requested frames. In order to make this integration efficient, the PFR Server has a defined set of services that can be called by the media application using a REST interface (in this architecture – via the BlackPearl plugin).
The interface supports three basic request calls and two status calls to monitor the progress of the requested execution:
For storing the file, the typical flow of execution would be:
Copy media file to NAS storage
Request creation of media index file
Monitor status for completion
Write file to BlackPearl Storage
For retrieving the partial file, the flow of execution would be:
Request File Byte Offsets based on timecode
Read specified byte range from BlackPearl to NAS
Request partial file creation
Monitor file creation status
Supply partial file to Media Application
Spectra’s development team has created an C# and Java SDKs to support the rapid development and integration of the PFR with BlackPearl Storage into existing M&E environments. The SDKs provide “wrapper” calls for the REST API calls. The C# and Java client documentation and code samples can be found at:
Spectra Logic’s BlackPearl is a gateway to our deep storage that uses an Application Programming Interface (API) for archive and restore operations. This API is used by our partners to integrate their software with BlackPearl. With the success of BlackPearl, the list of partner software clients is growing quickly. With the larger list comes a very important question: what should Spectra do to keep changes in BlackPearl code from affecting partner integrations? Retesting every release of BlackPearl code with every client version is obviously not feasible. Instead, Spectra Logic has chosen to tightly control and minimize API changes to prevent problems with partner integrations. To achieve this, we have established an API change review process that involves all stakeholders within the Spectra organization, as well as external partners and customers.
The release process:
-- Requires that every type of proposed change to an existing API call has a sponsor and is logged in the engineering feature tracking system.
-- The proposed change has to be reviewed and approved by both Spectra’s Engineering and Product Management departments before it becomes scheduled for development.
-- Even when approved, any changes to API with potential to affect partners and customers have to be scheduled with sufficient lead time to allow proper communications to and changes by all stakeholders, including customers, partners, and Spectra Logic.
-- Finally, the process restricts API changes which affect backward compatibility with existing clients to major releases of BlackPearl code (e.g. version 3.x to version 4.x), unless the change is deemed critical by the directors of both Engineering and Product Management departments.
-- New XML response elements and attributes are permitted, as they should be ignored by clients not taking advantage of a new response element. This behavior allows existing older clients to keep functioning with newer versions of API.
With all the controls built into the process, Spectra Logic aims to assure a stable and consistent API for interfacing with BlackPearl. At the same time, the process provides a means to expand the API to meet the needs of end users and support new applications and markets as they develop in the future.