Using the New BlackPearl “Staging Objects” Feature

Included in Spectra Logic’s BlackPearl 5.0 API is a new Stage Objects command. Most BlackPearl customers utilize tape storage, with many using tape as the final place where data will live. Meaning, over time, the data will age out of both the cache and temporary disk tier, leaving the only copy or copies of the data on tape. Due to the secure offline nature of tape, restore jobs which mount a tape cartridge will always have some latency as it will take time to retrieve a cartridge from a shelf inside of the tape library. This latency can compound into a rather long wait if one job, with a large set of objects and partial objects, spans across many tape cartridges.

This type of latency happens frequently in many environments, since their data sets can be quite large, potentially spanning tens or hundreds of tapes. For instance, in weather forecasting, weather data is gathered over time and archived daily to new tape. When a new climate model is set to be run on the supercomputer, the job requires a sub set of data from each day in the archived time period (potentially over the last 10 years), which is stored across hundreds of tapes.

If a large restore job requires mounting many tapes, this job can take a very long time to restore the data back onto the processing server or supercomputer, especially if there is a limit on drive availability. This waiting can waste time and money because data is sitting idle on expensive compute storage waiting for the rest of the data set to arrive before performing the analysis.

With the Stage Objects command, this same job can be given to BlackPearl and it will instead internally copy the data to either a disk tier, if available, otherwise it will copy the job to cache. Later, when the window of processing time is available, a normal bulk get job can restore the data from BlackPearl disk more efficiently and quickly, since the data is pulled in parallel without additional latency required mounting tapes.

Creating a Stage Objects job works just like creating a bulk get job, but instead of the operation being “START_BULK_GET”, it is instead “START_BULK_STAGE”.

Here is the HTTP request syntax:

PUT http[s]://{datapathDNSname}/_rest_/bucket/{bucket UUID or name}?operation=START_BULK_STAGE[&name={string}]​[&priority=URGENT|​HIGH|​NORMAL|​LOW]

The payload will include some number of objects:

<Objects
<Object Name=”{string}” Length=”{64-bit integer}”
Offset=”{64-bit integer}” Version_Id=”{string}”/>

</Objects>

The response is same as other bulk jobs:

<MasterObjectList
Aggregating=”TRUE|FALSE”
BucketName=”{string}”
CachedSizeInBytes=”{64-bit integer}”
ChunkClientProcessingOrderGuarantee=”IN_ORDER|NONE”
CompletedSizeInBytes=”{64-bit integer}”
EntirelyInCache=”TRUE|FALSE”
JobId=”{string}”
Naked=”TRUE|FALSE”
Name=”{string}”
OriginalSizeInBytes=”{64-bit integer}”
Priority=”CRITICAL|URGENT|HIGH|NORMAL|LOW|BACKGROUND”
RequestType=”GET”
StartDate=”YYYY-MM-DDThh:mm:ss.xxxZ”
Status=”IN_PROGRESS|COMPLETED|CANCELED”
UserId=”{string}”
UserName=”{string}”>
<Nodes>
<Node EndPoint=”{string}” Id=”{string}”/>
</Nodes>
<Objects
ChunkId=”{string}”
ChunkNumber=”{32-bit integer}”>
<Object Id=”{string}” InCache=”TRUE|FALSE”
Latest=”TRUE|FALSE” Length=”{64-bit integer}”
Name=”{string} “Offset=”{64-bit integer}”
VersionId=”{string}”/>

</Objects>

</MasterObjectList>

This Staging Objects operation is available in all 5.0+ SDKs for integration into your application. We encourage developers to use this new feature to pre-stage data for users of their applications. Contact the Developer Program Team if you have any questions or need assistance.


BlackPearl 5 Simulator Now Available

We have released a new BlackPearl simulator which is running the latest version of our BlackPearl 5.0 code, scheduled for release in late June. You can get the simulator on our Downloads page. SDKs for BlackPearl 5 are also available on this page. Make sure to carefully follow the simulator setup instructions. Contact the Developer Program Team if you need assistance.


Upcoming BlackPearl 5.0 Release Requires Client Recertification

BlackPearl’s next release, 5.0, is expected in June, and will introduce several important and exciting new features for our customers, including the following:

  • Intelligent Object Management – This features allows objects to self-heal if there is a problem detected with one copy and there are multiple copies available; allows tapes to be re-compacted; allows data to be migrated from one tape type to another; and allows objects to be staged from tape to disk for faster read access.
  • Versioning – BlackPearl will allow multiple versions of the same objects to be uploaded and saved

In order to implement these new features, changes to the BlackPearl API are required that could affect existing BlackPearl clients. With some major BlackPearl releases (3.0, 5.0, etc.), changes to the BlackPearl API are introduced to support new features and to remove features that have been deprecated. We do not take these API changes lightly, as we know that any change can affect the functionality of BlackPearl clients. We do everything we can to limit these changes while still allowing for the addition of new features.

The changes in 5.0 will require all BlackPearl clients to be updated, recertified, and retested before they can be used with BlackPearl 5.0. At a high level, here are some of the changes being made to the API that could affect existing clients:

Syntactic changes:

  • Removed fields in payload responses -- can cause errors when these entities are queried.
    • Tapes and pools no longer have storage domain ids -- they have storage domain member ids
    • Objects no longer have the old “version” field because the id is now considered the version
    • Data Policy no longer has “always replicate deletes” or “LTFS naming allowed” -- this functionality has been moved
  • Removed request parameters -- can cause errors if a customer tries to use them
    • The “import conflict resolution mode” query parameter for performing tape and pool imports has been removed
    • The following bucket attributes have been removed:
      •     DEFAULT_WRITE_OPTIMIZATION
      •     DEFAULT_GET_JOB_PRIORITY
      •     DEFAULT_PUT_JOB_PRIORITY
      •     OBJECT_SPANNING
      •     DEFAULT_CHECKSUM
      •     MIN_CLIENT_CHECKSUM
      •     MAX_CLIENT_CHECKSUM
    • Items in the previous section can no longer be filtered by their removed attributes. For example, “get tapes” cannot be filtered by “storage domain id”
  • XML payload attributes that are no longer valid:
    •   priority (now specified as a query parameter)
    •   chunk_client_processing_order_guarantee (now specified as a query parameter)
    •   write optimization (deprecated for xml payloads)
  • Changes to enums -- this includes things like new error message types that can cause errors when received by older clients
    • “Keep multiple versions” is a new versioning type.
    • “Auto compaction in progress” is a new tape type

Semantic Changes:

  • Requests that used to return a 503 “object not uploaded yet” now return a 404.
  • Objects not fully uploaded are no longer listed in an AWS-style “get bucket” command
  • Objects are now marked “latest” once they fully arrive in cache -- not when the job is created.
  • CacheAvailableRetryAfterInSeconds decreased from 300 seconds to 60 seconds -- this is how long Black Pearl recommends the client wait between checking for work to be done on a job
  • The “import conflict resolution mode” query parameter for performing tape and pool imports is still available as an optional parameter, but is ignored if sent.

You can view the full API difference comparison between BlackPearl 4.1 and 5.0. Note that the BlackPearl 5.0 API is still subject to change.

At the very least, clients will need to update to a new release of our SDKs, which will be available on our Downloads page. In some cases, code changes may be required to client integrations.

We have released the 5.0 beta SDKs for Java and .NET now. The Python, C, and Golang SDKs for 5.0 should be available shortly. We also have a BlackPearl running the 5.0 beta release that is available over the Internet. Please Contact the Developer Program Team when you wish to access it.

Testing and recertification will be required to ensure that your client works with BlackPearl 5.0. We will require you to redo the file transfer tests (archive and restore) specified in our Certification Program Test Plan. These tests will need to be done in our lab or at a facility with a BlackPearl and tape library.

Please Contact the Developer Program Team if you require assistance.


BlackPearl Eon Browser 2.1 Official Release

BlackPearl Eon Browser 2.1 has been officially released. The official release version is 2.1.6. There are many new features and bug fixes since the prior 2.0 release. We recommend users upgrade to 2.1.6 at their earliest convenience. You can get this release on our Downloads page. Contact the Developer Program Team if you have any questions or run into any problems.


Eon Browser 2.1 Beta Available for Testing

We have released a beta of the next version of the Eon Browser, version 2.1.4. You can download the Eon Browser beta on GitHub. A draft version of the new User Guide is also available. Note that this version should not be used in a production environment and will not be supported by Spectra Support. You can submit feedback via GitHub or by Contacting Us. We look forward to receiving your feedback.

 

 


BlackPearl SDK for Go is Now Available

Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We are pleased to announce that now we have an SDK for the Go programming language to simplify and accelerate BlackPearl client application development. In addition to Go, we currently support SDKs for Java, C#/.NET, C and Python. You can download the Go SDK and view the Go SDK documentation. This is a new language release, so please send us any feedback you might have on this SDK.


New BlackPearl 4.0 Simulator – Includes Simulated Tape Storage

We have released a new version of our BlackPearl simulator, version 4.0, to go with our BlackPearl 4.0 software release this week. In addition to having the latest BlackPearl code, this simulator now also includes a simulated tape library. Previous versions of the simulator only included storage of files in the BlackPearl disk cache. This new version allows files to be stored to cache and then to tape, providing a more realistic version of how a typical BlackPearl behaves when connected to a tape library. This also means that BlackPearl archive operations -- called “Jobs” -- can now reach a final completed state.

BlackPearl client developers, partners, and customers should find this new version of the simulator much more useful for their BlackPearl testing. Please Contact the BlackPearl Developer Program if you have any questions or feedback.


Getting Started with BlackPearl Partial File Restore Integration

In ​the Media & Entertainment world​, data files have reached very large sizes,​ particularly in cases of high resolution video ​that can exceed the one terabyte in size. In order to efficiently work with very large files, the media file processing is done in sections, with​ the end-user requesting content “snippets” based on timecodes. Object storage devices​​ that are used to store very large files are typically not aware of the timecode-to-byte relationship, and have no content awareness that’s necessary to extract and create partial media files.  In order to bridge the gap between time and bytes, BlackPearl has added a Partial File Restore (PFR) feature to enable the media processing application to efficiently retrieve a complete media file based on timecode offsets.

In​​ a typical Media Asset Management (MAM)/BlackPearl environment, the architecture would involve the following elements:

--        MAM – provides end user interface

--        BlackPearl MAM Plugin – handles all interactions between BlackPearl and MAM

--        BlackPearl – object storage

--        PFR Server – provides PFR processing services

--        NAS – network storage – permanent storage location for index files, temporary storage location for file processing

 

In the BlackPearl environment, there are two parts to the PFR processing of media files. The first part involves creating the media-aware index file for timecode/byte-offset information during file storage (writing) process. The second part of the process involves recalling a range of bytes that are based on a specified timecode, and creating complete, a self-contained media file that contains only the requested frames. In order to make this integration efficient, the PFR Server has a defined set of services that can be called by the media application using a REST interface (in this architecture – via the BlackPearl plugin).

The interface supports three basic request calls and two status calls to monitor the progress of the requested execution:

--        Request File Indexing
--        Request File Indexing Status
--        Request File Byte Offsets
--        Request Partial File Creation
--        Request Partial File Creation Status

For storing the file, the typical flow of execution would be:

  1. Copy media file to NAS storage
  2. Request creation of media index file
  3. Monitor status for completion
  4. Write file to BlackPearl Storage

For retrieving the partial file, the flow of execution would be:

  1. Request File Byte Offsets based on timecode
  2. Read specified byte range from BlackPearl to NAS
  3. Request partial file creation
  4. Monitor file creation status
  5. Supply partial file to Media Application

Spectra’s development team has created an C# and Java SDKs to support the rapid development and integration of the PFR with BlackPearl Storage into existing M&E environments. The SDKs provide “wrapper” calls for the REST API calls. The C# and Java client documentation and code samples can be found at:

https://github.com/SpectraLogic/tpfr_client

https://github.com/SpectraLogic/tpfr_java_client

PFR Users Guide provides the list of supported video file formats.

Quick Start Guide is a required reading to understand the services and locations that have to be setup in order to start PFR development.

PFR REST API definition is available for direct HTML development.

If you are considering creating a PFR integration with a BlackPearl environment, please contact the Spectra Logic Developer Program Team.


BlackPearl SDK for Python 3 Now Available

Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We provide these SDKs in Java, C#/.NET, C, and Python. Our initial Python SDK was built to be used with Python 2. We have now also released a Python SDK that is intended for Python 3.

You can download the Python 3 SDK now. Note that this is a beta release, so please send us any feedback you have on this SDK.

If you are not sure which Python version to use, read about Python 2 versus Python 3 on Python.org.


Minimizing Changes to the BlackPearl API

Spectra Logic’s BlackPearl is a gateway to our deep storage that uses an Application Programming Interface (API) for archive and restore operations. This API is used by our partners to integrate their software with BlackPearl. With the success of BlackPearl, the list of partner software clients is growing quickly. With the larger list comes a very important question: what should Spectra do to keep changes in BlackPearl code from affecting partner integrations? Retesting every release of BlackPearl code with every client version is obviously not feasible. Instead, Spectra Logic has chosen to tightly control and minimize API changes to prevent problems with partner integrations. To achieve this, we have established an API change review process that involves all stakeholders within the Spectra organization, as well as external partners and customers.

The release process:

-- Requires that every type of proposed change to an existing API call has a sponsor and is logged in the engineering feature tracking system.

-- The proposed change has to be reviewed and approved by both Spectra’s Engineering and Product Management departments before it becomes scheduled for development.

-- Even when approved, any changes to API with potential to affect partners and customers have to be scheduled with sufficient lead time to allow proper communications to and changes by all stakeholders, including customers, partners, and Spectra Logic.

-- Finally, the process restricts API changes which affect backward compatibility with existing clients to major releases of BlackPearl code (e.g. version 3.x to version 4.x), unless the change is deemed critical by the directors of both Engineering and Product Management departments.

-- New XML response elements and attributes are permitted, as they should be ignored by clients not taking advantage of a new response element. This behavior allows existing older clients to keep functioning with newer versions of API.

 

With all the controls built into the process, Spectra Logic aims to assure a stable and consistent API for interfacing with BlackPearl. At the same time, the process provides a means to expand the API to meet the needs of end users and support new applications and markets as they develop in the future.