Spectra BlackPearl platform is flexible, scalable architecture that can manage disk, tape and cloud storage.
In the latest release of the BlackPearl’s software version 5.4.2, Spectra have release their Attack Hardened software feature. This coincides with the released of the next generation of the BlackPearl hardware platform.
Spectra’s attack hardened software feature adds a range of different software features that help mitigate the risk of data loss and accelerates ransomware response. The attack hardening features include immutable snapshots, the automatic triggering of snapshots, Multi-Factor Authentication (MFA), snapshot change percentage detection along with others to help protect against an attack.
The next generation of BlackPearl hardware is a complete technology refresh of the BlackPearl hardware. It builds on the success of the current hardware platform, but it now offers customers more density, flexibility and speed. In the new hardware customers can to install up to 10 NVMe’s and 60 HDDs (8TB, 16TB or 20TB options available) directly the head unit. Spectra have also chosen to use AMD’s Epic family of chips to power the new generation of hardware.
Contact your local Spectra sales representative to know more.
In the Media & Entertainment world, data files have reached very large sizes, particularly in cases of high resolution video that can exceed the one terabyte in size. In order to efficiently work with very large files, the media file processing is done in sections, with the end-user requesting content “snippets” based on timecodes. Object storage devices that are used to store very large files are typically not aware of the timecode-to-byte relationship, and have no content awareness that’s necessary to extract and create partial media files. In order to bridge the gap between time and bytes, BlackPearl has added a Partial File Restore (PFR) feature to enable the media processing application to efficiently retrieve a complete media file based on timecode offsets.
In a typical Media Asset Management (MAM)/BlackPearl environment, the architecture would involve the following elements:
-- MAM – provides end user interface
-- BlackPearl MAM Plugin – handles all interactions between BlackPearl and MAM
-- BlackPearl – object storage
-- PFR Server – provides PFR processing services
-- NAS – network storage – permanent storage location for index files, temporary storage location for file processing
In the BlackPearl environment, there are two parts to the PFR processing of media files. The first part involves creating the media-aware index file for timecode/byte-offset information during file storage (writing) process. The second part of the process involves recalling a range of bytes that are based on a specified timecode, and creating complete, a self-contained media file that contains only the requested frames. In order to make this integration efficient, the PFR Server has a defined set of services that can be called by the media application using a REST interface (in this architecture – via the BlackPearl plugin).
The interface supports three basic request calls and two status calls to monitor the progress of the requested execution:
For storing the file, the typical flow of execution would be:
Copy media file to NAS storage
Request creation of media index file
Monitor status for completion
Write file to BlackPearl Storage
For retrieving the partial file, the flow of execution would be:
Request File Byte Offsets based on timecode
Read specified byte range from BlackPearl to NAS
Request partial file creation
Monitor file creation status
Supply partial file to Media Application
Spectra’s development team has created an C# and Java SDKs to support the rapid development and integration of the PFR with BlackPearl Storage into existing M&E environments. The SDKs provide “wrapper” calls for the REST API calls. The C# and Java client documentation and code samples can be found at:
Spectra Logic’s highest priority is protecting our customers’ data. BlackPearl and the tape libraries that sit behind it have a number of features to ensure that your data stays protected. We start data integrity as soon as data is ingested by BlackPearl. Client applications sending data to BlackPearl can pass in a checksum to ensure that the data arrives safely. A checksum acts as a fingerprint for the file and can be used to make sure that the file received by BlackPearl is the same file that the client thought it sent. BlackPearl accepts MD-5, CRC-32, CRC-32C, SHA-256, or SHA-512 checksums. The client provides the checksum type and value in the header of the API PUT operation:
BlackPearl uses the checksum provided for each file to ensure that the file it received is the same as the file the client sent. If the checksum does not match correctly with the file that was received, BlackPearl will return an error (400 – bad digest) to the client. If the client does not pass in a checksum with the file, BlackPearl will automatically create a checksum for the file. By default this checksum type will be MD-5, though the checksum type can be changed on the data policy settings on the bucket (a bucket is a top-level container for objects/files in BlackPearl). BlackPearl then stores the checksum, whether generated by the client or BlackPearl itself, both in its object database and with the file on tape.
Once stored by BlackPearl, the checksum can be used internally to make sure the file is still valid when it is recalled back from tape. BlackPearl will automatically do a checksum if the file is requested by a client (GET) and the file must come from tape. The checksum is done both as the file is coming from tape and after it has landed on the BlackPearl cache. BlackPearl will also provide the checksum value to the client so that the client can verify that it successfully received the file as well.
In some cases, when using the Bulk PUT operations, the client may be required to break the object into multiple “blobs” to upload it to BlackPearl. In this case, a checksum will be used for each blob uploaded to BlackPearl. The same is true when using Multi-Part Upload – each part of the file will have its own checksum. BlackPearl also supports Partial-File Restore, which is the ability to restore (GET) part of a file. With Partial-File Restore, a client can specify, for example, that it wants to retrieve the first 2GB of a 10GB file. In order to perform a checksum in this case, BlackPearl must first retrieve the entire file (or chunk) from tape. Once BlackPearl has completed the checksum on the file or chunk, it can send the partial file to the client.
Because BlackPearl may not always calculate the checksum for an entire file (because it may be broken up into multiple pieces), developers may want to have their clients calculate the entire file checksum itself. This value could then be stored in the file’s metadata when uploaded to BlackPearl. When the file is later retrieved, the client could calculate the checksum again and compare the values.
Read our quarterly Developer Program Newsletter, which was just released today. Learn about BlackPearl 1.2, our Developer Summit, the Java CLI, our training videos, and more.
Tuesday, October 20, 2015 9:00 AM Mountain Time (15:00 UTC), WebEx
Join us for Spectra Logic’s inaugural BlackPearl Developer Summit, a virtual conference for current and potential Spectra Logic developers. You’ll get product updates from our CEO and BlackPearl product manager, and you will learn how these new features will help customers and developers. You will learn how to build a Spectra S3 client for BlackPearl, our private cloud gateway to our tape and disk storage systems. You will see how one of our partners developed a client and watch it in action. And you will get to ask questions to our BlackPearl Engineering team. Don’t miss it! Learn more and register at spectralogic.com/developerconference
