Spectra StorCycle Now Has Exposed API

Spectra StorCycle is a storage lifecycle management solution that allows users to identify inactive data, move or copy it to a lower-cost tier of storage, protect it to ensure that it is safe, and easily restore it as needed. With the 3.3 release of StorCycle in January 2021, Spectra Logic has exposed some of the API commands of StorCycle to allow our customers to build custom integrations to StorCycle.

The exposed commands currently include:

  • Authentication
  • Create and manage storage locations (both sources and targets)
  • Create and manage scan, migrate/archive, and restore operations

These commands might be used for actions such as bulk creation of storage locations, scripting of migrations that first require actions of other systems, and allowing user restores through a separate web portal.

We have provided a full set of documentation for each command, as well as code samples in Python.

Learn more about StorCycle API integrations


How to Use Python 3 BlackPearl Documentation

Spectra BlackPearl includes an Application Program Interface (API) that allows applications to interact with BlackPearl. Spectra Logic has created a number of Software Development Kits (SDKs) in various programming languages, including Python 3, which act as an overlay to the API and make it easier to interact with the API in the selected programming language.

For the Python 3 SDK, documentation has been provided that explains how to use the commands. This documentation is automatically generated from the code. While the documentation is very helpful, it can be difficult to interpret. The example below should assist you in using the documentation.

The first place to start is in the API documentation. You should first review this API documentation and determine which command you wish to use. As an example, perhaps you want to determine the BlackPearl system health. This would be done with the Verify System Health API command. The API documentation for this page includes the command syntax and the request and response parameters, as shown below.

You next need to find the corresponding Python components for this command in the Python documentation. As the API command in the API documentation is titled “Verify System Health”, we now first want to search the Python documentation page for “VerifySystemHealth” (no spaces). You can usually use Ctrl+F or Command+F to search a web page for text. Your search should take you to the part of the page with the associated Python classes for this command, as shown below.

We now need to search the Python documentation page again for the associated function call. Again, as the API command in the API documentation is titled “Verify System Health”, we this time want to search the Python documentation page for “verify_system_health” (all lower case, replace spaces with underscores). We can now see the function name for this API command, as shown below:

Now we will create the Python code to call the Verify System Health command. We first need to make sure that we import the SDK library and that we create a “client” object for the BlackPearl we are calling:

You will need to provide the actual credentials for your BlackPearl system.

Next we will create the Python code for the Verify System Health call:

We can review the various components of the code:

  • getHealthResponse – this is an arbitrary variable name that you choose
  • client – this is the client variable from above when we created the client object
  • verify_system_health_spectra_s3 – this is the function name we found on the Python documentation page
  • ds3.VerifySystemHealthSpectraS3Request() – ds3 is the SDK code library; VerifySystemHealthSpectraS3Request is the class name we found on the Python documentation page
  • DatabaseFilesystemFreeSpace – this is the response parameter we found on the API documentation page
  • MsRequiredToVerifyDataPlannerHealth – this is the response parameter we found on the API documentation page

Hopefully this example will help you use the documentation to create Python scripts and applications for BlackPearl.


Java CLI Lists of Objects Now Goes Beyond 1000

The Java CLI “get_bucket” command can provide a list of objects in a BlackPearl bucket. However, up until now, it could only provide up to 1000 objects in the list. If the bucket contained more than 1000 objects, the command could not list all of the objects.

The same was true for the  “get_objects_on_tape” command. It only listed up to 1000 objects on a tape, even if the tape actually contained more than 1000 objects.

All objects will now be listed using these commands, not just the first 1000.

You can download the latest release here:

https://github.com/SpectraLogic/ds3_java_cli/releases/tag/v5.1.2


RioBroker “Relationship” Feature to Be Removed in 3.0 Release

Spectra RioBroker is a data mover that allows our partners and customers to build solutions that move data into and out of BlackPearl. RioBroker includes a robust API to allow software applications to manage this data movement process.

RioBroker’s API currently includes a “Relationships” feature. The Relationships feature allows a user to specify an ID and link many objects to that ID. An application could then look and find all related files through the relationship ID. Relationships has not seen use by users and was implemented in order to facilitate the linking or aggregating of multiple objects together. In an effort to better solve the problem we are going to remove this feature, and longer term we will provide a solution better tailored to solving the object linking problem.

Starting with the current release of RioBroker, 2.1, Relationships are deprecated, and if a user or integration is using it we recommend you start removing it now. In the following release, 3.0, due in September 2020, the feature will be removed.

For any immediate needs, if having related files being easily identifiable is needed, then we recommend using a common prefix for object names. So if your object name is video.mfx, and has a related file, audio.mxf, then we recommend storing the objects with a name like, uniqueId/video.mfx and uniqueId/audio.mxf. We are currently evaluating how best to link files together into a single logical object for the longer-term solution.

Please Contact the Developer Program if you have any questions or concerns.


No More Releases for Python 2 SDK

The Python Software Foundation has announced the “Sunsetting” of Python 2. Spectra Logic currently provides a BlackPearl Software Development Kit (SDK) using Python 2. With the announced sunsetting of Python 2, Spectra Logic will no longer provide future releases of the Python 2 SDK to match future releases of BlackPearl. This means that any changes or updates to the BlackPearl API will not be available via the Python 2 SDK. The Python 2 SDK should continue to work with future versions of BlackPearl (5.2, 5.3, etc.) until the 6.0 release, at which time there will be changes to the BlackPearl API that will cause the Python 2 SDK to no longer work. There are no plans for a BlackPearl 6.0 release at this time. Whole number changes in the BlackPearl release number (e.g. 5.x, 6.x) indicate breaking API changes.

Customers should work to update to the Python 3 SDK as soon as possible.

Spectra Logic may choose to address bug fixes in the latest Python 2 SDK on a case-by-case basis.

Please contact the Spectra Logic Developer program if you have any questions or concerns.


Restoring Large Files in Windows with BlackPearl – Sparse Files

Spectra Logic provides Software Development Kits (SDKs) for BlackPearl in several different programming languages to make it easier for our partners and customers to create integrations to BlackPearl. Some of the BlackPearl SDKs are used in Windows environments, particularly the Java and .NET SDKs. An issue was recently discovered in the way that these SDKs write large data blocks (64GB+) restored from BlackPearl on to disk.

In Windows, by default, when a file is created, Windows will initially reserve the space for the file on disk by filling the file with zeros. It then later replaces those zeros with the actual file byte content as it is received. When the files are 64GB or larger, it can take a very long time to write all of the zeros. In that time the file transfer from BlackPearl can time out, causing the SDK to have to resubmit its request to BlackPearl to restore the file. This resubmission causes the overall transfer process to take longer than it would if the resubmission did not occur. And in extreme cases with extremely large files (1TB+), the transfer can permanently fail.

To avoid this time out issue, the zeros should not be initially written in the file. This can be done by specifying that Windows Sparse Files be used. With Windows sparse files, per Microsoft, “the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data”. Sparse files are therefore initially written much more quickly than non-sparse files.

We have updated our Java and .NET SDKs to use sparse files on restores. Partners and customers using these SDKs should update to the latest versions to ensure that they have this sparse file fix. However, the fixes in these SDKs will only be helpful if the “FileHelpers” class is being used, in which case the SDKs will be writing the restored files. For those organizations that are using these SDKs but have written their own code to write the restored files, they must implement sparse files themselves in their own code.

You can see examples of how Spectra Logic implemented sparse files in the SDKs:

Please Contact the Spectra Developer Program Team  if you need assistance with implementing sparse files with BlackPearl.


A New Method of BlackPearl Integration – Spectra RioBroker

To date, hundreds of Spectra Logic customers have implemented our BlackPearl® Converged Storage System to affordably store data long-term. Spectra’s BlackPearl is a purpose-built storage platform that integrates directly with ”data mover” software applications to simplify workflows and seamlessly manage large volumes of data to multiple storage targets. BlackPearl includes an S3-like interface to move data into its object storage on the cloud, disk, and tape. In order to help even more customers meet their long-term storage needs, Spectra has recently introduced another, easier system for transferring files into BlackPearl. This new system is called Spectra RioBroker. The announcement, made yesterday, can be read here.

As a software front-end to BlackPearl, Spectra RioBroker acts as a data mover for applications that wish to move data into the BlackPearl object storage gateway. The typical system architecture is shown in the diagram below.

There are several advantages to using Spectra RioBroker instead of the direct BlackPearl i­nterface:

  • Enables easier client development for partners with a simple abstraction layer over BlackPearl interface
  • Allows more clients and applications to share BlackPearl object storage resources in parallel at ever higher performance
  • Provides remote input/output capabilities to multiple Spectra BlackPearl Converged Storage servers
  • Brokers data stream input and output between multiple sources and destinations
  • Provides for enterprise-level high availability*
  • Delivers ultra-high performance with clustering capabilities*
  • Facilitates seamless content migration from legacy storage software to a modern solution

A perfect use case example for Spectra RioBroker is for Media Asset Management (MAM) software typically used in the Media and Entertainment industry. Spectra’s MAM partners have built integrations to Spectra RioBroker so that their customers can archive and restore files directly from the MAM interface without the files passing through the MAM server. Now that the software is released, we expect many more partners and customers to build integrations.

The Spectra RioBroker software application is provided to Spectra customers at no cost. The software is currently available for Windows servers, and a Linux version will be released in the future. Both partners and customers can utilize the free tools on Spectra’s Developer Program Website to build their own integrations. The website includes all the tools needed to build an integration, including the Spectra RioBroker installer, a BlackPearl simulator, an SDK, code examples, documentation and more. Spectra RioBroker, like BlackPearl, uses a RESTful API with very simple commands. A basic integration to Rio Broker requires only three API commands – Archive, Restore, and Check Job Status.

While Spectra RioBroker will become a popular method to integrate to BlackPearl, direct integration to BlackPearl will continue to be logical for certain use cases. Spectra Logic will continue to support this direct integration method.

*Available in future release of Spectra RioBroker


Using the New BlackPearl “Staging Objects” Feature

Included in Spectra Logic’s BlackPearl 5.0 API is a new Stage Objects command. Most BlackPearl customers utilize tape storage, with many using tape as the final place where data will live. Meaning, over time, the data will age out of both the cache and temporary disk tier, leaving the only copy or copies of the data on tape. Due to the secure offline nature of tape, restore jobs which mount a tape cartridge will always have some latency as it will take time to retrieve a cartridge from a shelf inside of the tape library. This latency can compound into a rather long wait if one job, with a large set of objects and partial objects, spans across many tape cartridges.

This type of latency happens frequently in many environments, since their data sets can be quite large, potentially spanning tens or hundreds of tapes. For instance, in weather forecasting, weather data is gathered over time and archived daily to new tape. When a new climate model is set to be run on the supercomputer, the job requires a sub set of data from each day in the archived time period (potentially over the last 10 years), which is stored across hundreds of tapes.

If a large restore job requires mounting many tapes, this job can take a very long time to restore the data back onto the processing server or supercomputer, especially if there is a limit on drive availability. This waiting can waste time and money because data is sitting idle on expensive compute storage waiting for the rest of the data set to arrive before performing the analysis.

With the Stage Objects command, this same job can be given to BlackPearl and it will instead internally copy the data to either a disk tier, if available, otherwise it will copy the job to cache. Later, when the window of processing time is available, a normal bulk get job can restore the data from BlackPearl disk more efficiently and quickly, since the data is pulled in parallel without additional latency required mounting tapes.

Once the data is restored to disk, the staged objects will stay on the disk as dictated by the disk policy. For example, if the data is restored to ArcticBlue disk that has a 3-month retention policy policy, the data will stay on the disk for at least 3 months. If it’s restored to disk cache, it will stay on cache for as long as the cache can keep it there until it has to make room for other files.

Creating a Stage Objects job works just like creating a bulk get job, but instead of the operation being “START_BULK_GET”, it is instead “START_BULK_STAGE”.

Here is the HTTP request syntax:

PUT http[s]://{datapathDNSname}/_rest_/bucket/{bucket UUID or name}?operation=START_BULK_STAGE[&name={string}]​[&priority=URGENT|​HIGH|​NORMAL|​LOW]

The payload will include some number of objects:

<Objects
<Object Name=”{string}” Length=”{64-bit integer}”
Offset=”{64-bit integer}” Version_Id=”{string}”/>

</Objects>

The response is same as other bulk jobs:

<MasterObjectList
Aggregating=”TRUE|FALSE”
BucketName=”{string}”
CachedSizeInBytes=”{64-bit integer}”
ChunkClientProcessingOrderGuarantee=”IN_ORDER|NONE”
CompletedSizeInBytes=”{64-bit integer}”
EntirelyInCache=”TRUE|FALSE”
JobId=”{string}”
Naked=”TRUE|FALSE”
Name=”{string}”
OriginalSizeInBytes=”{64-bit integer}”
Priority=”CRITICAL|URGENT|HIGH|NORMAL|LOW|BACKGROUND”
RequestType=”GET”
StartDate=”YYYY-MM-DDThh:mm:ss.xxxZ”
Status=”IN_PROGRESS|COMPLETED|CANCELED”
UserId=”{string}”
UserName=”{string}”>
<Nodes>
<Node EndPoint=”{string}” Id=”{string}”/>
</Nodes>
<Objects
ChunkId=”{string}”
ChunkNumber=”{32-bit integer}”>
<Object Id=”{string}” InCache=”TRUE|FALSE”
Latest=”TRUE|FALSE” Length=”{64-bit integer}”
Name=”{string} “Offset=”{64-bit integer}”
VersionId=”{string}”/>

</Objects>

</MasterObjectList>

This Staging Objects operation is available in all 5.0+ SDKs for integration into your application. We encourage developers to use this new feature to pre-stage data for users of their applications. Contact the Developer Program Team if you have any questions or need assistance.


BlackPearl 5 Simulator Now Available

We have released a new BlackPearl simulator which is running the latest version of our BlackPearl 5.0 code, scheduled for release in late June. You can get the simulator on our Downloads page. SDKs for BlackPearl 5 are also available on this page. Make sure to carefully follow the simulator setup instructions. Contact the Developer Program Team if you need assistance.


Upcoming BlackPearl 5.0 Release Requires Client Recertification

BlackPearl’s next release, 5.0, is expected in June, and will introduce several important and exciting new features for our customers, including the following:

  • Intelligent Object Management – This features allows objects to self-heal if there is a problem detected with one copy and there are multiple copies available; allows tapes to be re-compacted; allows data to be migrated from one tape type to another; and allows objects to be staged from tape to disk for faster read access.
  • Versioning – BlackPearl will allow multiple versions of the same objects to be uploaded and saved

In order to implement these new features, changes to the BlackPearl API are required that could affect existing BlackPearl clients. With some major BlackPearl releases (3.0, 5.0, etc.), changes to the BlackPearl API are introduced to support new features and to remove features that have been deprecated. We do not take these API changes lightly, as we know that any change can affect the functionality of BlackPearl clients. We do everything we can to limit these changes while still allowing for the addition of new features.

The changes in 5.0 will require all BlackPearl clients to be updated, recertified, and retested before they can be used with BlackPearl 5.0. At a high level, here are some of the changes being made to the API that could affect existing clients:

Syntactic changes:

  • Removed fields in payload responses -- can cause errors when these entities are queried.
    • Tapes and pools no longer have storage domain ids -- they have storage domain member ids
    • Objects no longer have the old “version” field because the id is now considered the version
    • Data Policy no longer has “always replicate deletes” or “LTFS naming allowed” -- this functionality has been moved
  • Removed request parameters -- can cause errors if a customer tries to use them
    • The “import conflict resolution mode” query parameter for performing tape and pool imports has been removed
    • The following bucket attributes have been removed:
      •     DEFAULT_WRITE_OPTIMIZATION
      •     DEFAULT_GET_JOB_PRIORITY
      •     DEFAULT_PUT_JOB_PRIORITY
      •     OBJECT_SPANNING
      •     DEFAULT_CHECKSUM
      •     MIN_CLIENT_CHECKSUM
      •     MAX_CLIENT_CHECKSUM
    • Items in the previous section can no longer be filtered by their removed attributes. For example, “get tapes” cannot be filtered by “storage domain id”
  • XML payload attributes that are no longer valid:
    •   priority (now specified as a query parameter)
    •   chunk_client_processing_order_guarantee (now specified as a query parameter)
    •   write optimization (deprecated for xml payloads)
  • Changes to enums -- this includes things like new error message types that can cause errors when received by older clients
    • “Keep multiple versions” is a new versioning type.
    • “Auto compaction in progress” is a new tape type

Semantic Changes:

  • Requests that used to return a 503 “object not uploaded yet” now return a 404.
  • Objects not fully uploaded are no longer listed in an AWS-style “get bucket” command
  • Objects are now marked “latest” once they fully arrive in cache -- not when the job is created.
  • CacheAvailableRetryAfterInSeconds decreased from 300 seconds to 60 seconds -- this is how long Black Pearl recommends the client wait between checking for work to be done on a job
  • The “import conflict resolution mode” query parameter for performing tape and pool imports is still available as an optional parameter, but is ignored if sent.

You can view the full API difference comparison between BlackPearl 4.1 and 5.0. Note that the BlackPearl 5.0 API is still subject to change.

At the very least, clients will need to update to a new release of our SDKs, which will be available on our Downloads page. In some cases, code changes may be required to client integrations.

We have released the 5.0 beta SDKs for Java and .NET now. The Python, C, and Golang SDKs for 5.0 should be available shortly. We also have a BlackPearl running the 5.0 beta release that is available over the Internet. Please Contact the Developer Program Team when you wish to access it.

Testing and recertification will be required to ensure that your client works with BlackPearl 5.0. We will require you to redo the file transfer tests (tests 5 through 8) specified in our Certification Program Test Plan. These tests will need to be done in our lab or at a facility with a BlackPearl and tape library, and the tests results need to be approved by Spectra Logic in order to receive certification for BlackPearl 5.

Please Contact the Developer Program Team if you require assistance.