Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We are pleased to announce that now we have an SDK for the Go programming language to simplify and accelerate BlackPearl client application development. In addition to Go, we currently support SDKs for Java, C#/.NET, C and Python. You can download the Go SDK and view the Go SDK documentation. This is a new language release, so please send us any feedback you might have on this SDK.
We have released a new version of our BlackPearl simulator, version 4.0, to go with our BlackPearl 4.0 software release this week. In addition to having the latest BlackPearl code, this simulator now also includes a simulated tape library. Previous versions of the simulator only included storage of files in the BlackPearl disk cache. This new version allows files to be stored to cache and then to tape, providing a more realistic version of how a typical BlackPearl behaves when connected to a tape library. This also means that BlackPearl archive operations -- called “Jobs” -- can now reach a final completed state.
BlackPearl client developers, partners, and customers should find this new version of the simulator much more useful for their BlackPearl testing. Please Contact the BlackPearl Developer Program if you have any questions or feedback.
BlackPearl has seen very wide adoption across various storage system and across multiple use cases. To meet the requirements of many user groups, Spectra’s Engineering Team has been enhancing the features and interface of the product, while at the same time postponing any changes that could affect existing software clients. When the changes are made, we try to limit them to major releases, and provide our partners with as much notice as possible to allow time for code development and release. With the upcoming major release of BlackPearl’s code version 4.0 in August, we are including the following changes that could affect existing BlackPearl’s software clients:
Bucket naming conventions -- For Cloud Out operations with Amazon AWS or Microsoft Azure, the BlackPearl bucket name will have to follow cloud provider requirements. BlackPearl will attempt to create the bucket on the cloud storage provider with the name that is an exact match of the name that is specified in BlackPearl. If the bucket name does not follow the rules of the specific cloud service provider, an error will be returned and will indicate that that bucket could not be created and the error code(s) given by the cloud service provider will be passed along. (In previous versions of BlackPearl code, BlackPearl used only lower case letters to avoid upper case incompatibilities.)
Optional “Folder” parameter in “GetObjectsDetailsRequest” and “GetObjectsWithFullDetailsRequest” calls has been removed.
Failure code handling has been changed in the following error handlers: GetPoolFailures, GetTapeFailures, GetAzureTargetFailures, GetDS3TargetFailure, & GetS3TargetFailure.
File metadata requirement -- If a file is broken up into multiple parts or “blobs” when being sent to BlackPearl, the object metadata needs to be sent by the client application with each part/blob. Previously clients were only required to send metadata on the first part/blob. For clients using the C#/.NET or Java Software Development Kits (SDKs), this change will be handled automatically through the built-in helper classes, so simply updating to the latest 4.0 SDK will address the change. Clients using the Python or C SDK and clients that interact with the API directly (those that do not use our SDKs) will have to modify code to provide metadata with every blob in the transfer in order to work with next major release of BlackPearl code (version 5.0), expected early in 2018.
The Data Policy fields of “always_replicate_deletes” (request parameter) and “LtfsObjectNamingAllowed” (response parameter) are being deprecated and will be removed from the Data Policy in next major release of BlackPearl code (version 5.0), expected early in 2018.
We have provided pre-release versions of the BlackPearl version 4.0 SDKs to support code updating. These SDKs can be downloaded at the following locations:
We have also made a BlackPearl version 4.0 beta system available in Spectra’s lab that can be accessed over the Internet to support testing activities ahead of version 4.0 code release. To access this BlackPearl, please Contact us. It is strongly recommended that any client changes are certified/re-certified following our self-certification program
If you use BlackPearl SDKs in your client, even if you are not concerned about the changes above, you will still want to update your client to the latest 4.0 SDKs, and re-verify put/get functionality as outlined in BlackPearl Certification Test Plan sections 7.5-7.8.
Please Contact Us if you need assistance with upgrading your BlackPearl client to support the 4.0 release. We would like to ensure that existing BlackPearl clients work well with BlackPearl version 4.0.
In the Media & Entertainment world, data files have reached very large sizes, particularly in cases of high resolution video that can exceed the one terabyte in size. In order to efficiently work with very large files, the media file processing is done in sections, with the end-user requesting content “snippets” based on timecodes. Object storage devices that are used to store very large files are typically not aware of the timecode-to-byte relationship, and have no content awareness that’s necessary to extract and create partial media files. In order to bridge the gap between time and bytes, BlackPearl has added a Partial File Restore (PFR) feature to enable the media processing application to efficiently retrieve a complete media file based on timecode offsets.
In a typical Media Asset Management (MAM)/BlackPearl environment, the architecture would involve the following elements:
-- MAM – provides end user interface
-- BlackPearl MAM Plugin – handles all interactions between BlackPearl and MAM
-- BlackPearl – object storage
-- PFR Server – provides PFR processing services
-- NAS – network storage – permanent storage location for index files, temporary storage location for file processing
In the BlackPearl environment, there are two parts to the PFR processing of media files. The first part involves creating the media-aware index file for timecode/byte-offset information during file storage (writing) process. The second part of the process involves recalling a range of bytes that are based on a specified timecode, and creating complete, a self-contained media file that contains only the requested frames. In order to make this integration efficient, the PFR Server has a defined set of services that can be called by the media application using a REST interface (in this architecture – via the BlackPearl plugin).
The interface supports three basic request calls and two status calls to monitor the progress of the requested execution:
For storing the file, the typical flow of execution would be:
Copy media file to NAS storage
Request creation of media index file
Monitor status for completion
Write file to BlackPearl Storage
For retrieving the partial file, the flow of execution would be:
Request File Byte Offsets based on timecode
Read specified byte range from BlackPearl to NAS
Request partial file creation
Monitor file creation status
Supply partial file to Media Application
Spectra’s development team has created an C# and Java SDKs to support the rapid development and integration of the PFR with BlackPearl Storage into existing M&E environments. The SDKs provide “wrapper” calls for the REST API calls. The C# and Java client documentation and code samples can be found at:
Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We provide these SDKs in Java, C#/.NET, C, and Python. Our initial Python SDK was built to be used with Python 2. We have now also released a Python SDK that is intended for Python 3.
Spectra Logic’s BlackPearl is a gateway to our deep storage that uses an Application Programming Interface (API) for archive and restore operations. This API is used by our partners to integrate their software with BlackPearl. With the success of BlackPearl, the list of partner software clients is growing quickly. With the larger list comes a very important question: what should Spectra do to keep changes in BlackPearl code from affecting partner integrations? Retesting every release of BlackPearl code with every client version is obviously not feasible. Instead, Spectra Logic has chosen to tightly control and minimize API changes to prevent problems with partner integrations. To achieve this, we have established an API change review process that involves all stakeholders within the Spectra organization, as well as external partners and customers.
The release process:
-- Requires that every type of proposed change to an existing API call has a sponsor and is logged in the engineering feature tracking system.
-- The proposed change has to be reviewed and approved by both Spectra’s Engineering and Product Management departments before it becomes scheduled for development.
-- Even when approved, any changes to API with potential to affect partners and customers have to be scheduled with sufficient lead time to allow proper communications to and changes by all stakeholders, including customers, partners, and Spectra Logic.
-- Finally, the process restricts API changes which affect backward compatibility with existing clients to major releases of BlackPearl code (e.g. version 3.x to version 4.x), unless the change is deemed critical by the directors of both Engineering and Product Management departments.
-- New XML response elements and attributes are permitted, as they should be ignored by clients not taking advantage of a new response element. This behavior allows existing older clients to keep functioning with newer versions of API.
With all the controls built into the process, Spectra Logic aims to assure a stable and consistent API for interfacing with BlackPearl. At the same time, the process provides a means to expand the API to meet the needs of end users and support new applications and markets as they develop in the future.
When BlackPearl writes data to tape, it uses the open Linear Tape File System (LTFS) file format. Because of this LTFS support, Spectra Logic was able to add the ability to import non-BlackPearl or “foreign” LTFS tapes to BlackPearl. This is useful for any customer that receives LTFS-formatted tapes from another source and wishes to read those same tapes in the BlackPearl environment. This workflow is particularly common in the Media and Entertainment industry as a way to transfer video files from one group to another. Since every application may utilize the open LTFS format in a different manner when writing data to tapes, it is important that the user verify the various LTFS tape formats will be properly imported for read only use in the BlackPearl environment.
BlackPearl will support the import of foreign LTFS tapes in version 3.5, which is due out in Q1 of 2017. Importing can be done manually via the BlackPearl web management interface or via an external application that calls the BlackPearl API. The “import” process we discuss herein assumes that the foreign LTFS tapes have already been physically imported into the tape library partition to which BlackPearl is connected. Additionally, when a foreign LTFS tape is added to a BlackPearl tape partition, the write-protect switch on the tape cartridge must be set to “read only” before BlackPearl will allow the tape to be imported. We strongly recommend that the tape be kept in read-only mode so that BlackPearl will not be able to modify the tape in any way. Also note that the API import commands must be initiated by the administrator or “spectra” user.
To manually import foreign LTFS tapes via the web management interface of BlackPearl, users will go to the Tape Management page (Status > Tape Management), click on the tape(s) to be imported, and then go to the Action menu and select Import Foreign Tape.
The tapes can also be imported via an external application that calls the BlackPearl API. While there are several possible workflows that can be taken to import the tapes into BlackPearl by an application, this is one we are currently recommending to our partners:
Application calls API command Get Tapes to get a list of all tapes and their state. Tapes with a state of LTFS_WITH_FOREIGN_DATA should be recorded along with their barcodes.
Application calls API command Raw Import All Tapes to start the process of importing the foreign LTFS tapes. Make sure to make this call using the admin user “spectra”. The application must provide a bucket name in which to import the files on the foreign LTFS tapes. Alternatively the application can call Raw Import Tape to import individual foreign LTFS tapes based on the barcodes recorded from Step 1 above. Again, each time this is called, a bucket name will need to be provided and using this method can allow the application to import different tapes into different buckets if desired.
Application should periodically call API command Get Tapes again to check the state of all tapes. Once there are no more tapes with a state of LTFS Foreign, it means all foreign LTFS tapes have been successfully imported.
Application calls API command Get Physical Placement for Object Parts on Tape to get a list of all files that are on each LTFS tape, using the barcodes recorded in Step 1 above as the input parameter for this call. The application can then read each of these files if needed, such as to create video proxies or to copy them to another location in BlackPearl. If the application doesn’t need to read the files, it can simply record the list of file names.
If the application no longer needs the files on the foreign LTFS tapes, for example because it has read the files and copied them to another bucket in BlackPearl, it can then issue an Eject Tape command for each tape.
As you can see, it is quite easy for applications to work with foreign LTFS tapes in BlackPearl. It is important to remember that the user and/or developer need to confirm that the user’s LTFS tapes can be read by BlackPearl by testing them out. Also, note that BlackPearl will not generate a checksum for the files on foreign LTFS tapes as it does for other files that are imported into BlackPearl. We also don’t currently support importing foreign LTFS tapes that use “tape spanning”, or spanning one file across multiple tapes.
Foreign LTFS tape import is supported in released Java SDK 3.4.0-RC1, and it will be supported by any BlackPearl hardware with code 3.5 and higher. We will soon (in the next two months) also have the foreign LTFS tape import operations available in our other SDKs (C#/.NET, Python, C). We will also soon update our BlackPearl API documentation to include the new foreign LTFS tape import operations.