Spectra StorCycle Now Has Exposed API

Spectra StorCycle is a storage lifecycle management solution that allows users to identify inactive data, move or copy it to a lower-cost tier of storage, protect it to ensure that it is safe, and easily restore it as needed. With the 3.3 release of StorCycle in January 2021, Spectra Logic has exposed some of the API commands of StorCycle to allow our customers to build custom integrations to StorCycle.

The exposed commands currently include:

  • Authentication
  • Create and manage storage locations (both sources and targets)
  • Create and manage scan, migrate/archive, and restore operations

These commands might be used for actions such as bulk creation of storage locations, scripting of migrations that first require actions of other systems, and allowing user restores through a separate web portal.

We have provided a full set of documentation for each command, as well as code samples in Python.

Learn more about StorCycle API integrations

Java CLI Lists of Objects Now Goes Beyond 1000

The Java CLI “get_bucket” command can provide a list of objects in a BlackPearl bucket. However, up until now, it could only provide up to 1000 objects in the list. If the bucket contained more than 1000 objects, the command could not list all of the objects.

The same was true for the  “get_objects_on_tape” command. It only listed up to 1000 objects on a tape, even if the tape actually contained more than 1000 objects.

All objects will now be listed using these commands, not just the first 1000.

You can download the latest release here:


A New Method of BlackPearl Integration – Spectra RioBroker

To date, hundreds of Spectra Logic customers have implemented our BlackPearl® Converged Storage System to affordably store data long-term. Spectra’s BlackPearl is a purpose-built storage platform that integrates directly with ”data mover” software applications to simplify workflows and seamlessly manage large volumes of data to multiple storage targets. BlackPearl includes an S3-like interface to move data into its object storage on the cloud, disk, and tape. In order to help even more customers meet their long-term storage needs, Spectra has recently introduced another, easier system for transferring files into BlackPearl. This new system is called Spectra RioBroker. The announcement, made yesterday, can be read here.

As a software front-end to BlackPearl, Spectra RioBroker acts as a data mover for applications that wish to move data into the BlackPearl object storage gateway. The typical system architecture is shown in the diagram below.

There are several advantages to using Spectra RioBroker instead of the direct BlackPearl i­nterface:

  • Enables easier client development for partners with a simple abstraction layer over BlackPearl interface
  • Allows more clients and applications to share BlackPearl object storage resources in parallel at ever higher performance
  • Provides remote input/output capabilities to multiple Spectra BlackPearl Converged Storage servers
  • Brokers data stream input and output between multiple sources and destinations
  • Provides for enterprise-level high availability*
  • Delivers ultra-high performance with clustering capabilities*
  • Facilitates seamless content migration from legacy storage software to a modern solution

A perfect use case example for Spectra RioBroker is for Media Asset Management (MAM) software typically used in the Media and Entertainment industry. Spectra’s MAM partners have built integrations to Spectra RioBroker so that their customers can archive and restore files directly from the MAM interface without the files passing through the MAM server. Now that the software is released, we expect many more partners and customers to build integrations.

The Spectra RioBroker software application is provided to Spectra customers at no cost. The software is currently available for Windows servers, and a Linux version will be released in the future. Both partners and customers can utilize the free tools on Spectra’s Developer Program Website to build their own integrations. The website includes all the tools needed to build an integration, including the Spectra RioBroker installer, a BlackPearl simulator, an SDK, code examples, documentation and more. Spectra RioBroker, like BlackPearl, uses a RESTful API with very simple commands. A basic integration to Rio Broker requires only three API commands – Archive, Restore, and Check Job Status.

While Spectra RioBroker will become a popular method to integrate to BlackPearl, direct integration to BlackPearl will continue to be logical for certain use cases. Spectra Logic will continue to support this direct integration method.

*Available in future release of Spectra RioBroker

BlackPearl 5 Simulator Now Available

We have released a new BlackPearl simulator which is running the latest version of our BlackPearl 5.0 code, scheduled for release in late June. You can get the simulator on our Downloads page. SDKs for BlackPearl 5 are also available on this page. Make sure to carefully follow the simulator setup instructions. Contact the Developer Program Team if you need assistance.

Upcoming BlackPearl 5.0 Release Requires Client Recertification

BlackPearl’s next release, 5.0, is expected in June, and will introduce several important and exciting new features for our customers, including the following:

  • Intelligent Object Management – This features allows objects to self-heal if there is a problem detected with one copy and there are multiple copies available; allows tapes to be re-compacted; allows data to be migrated from one tape type to another; and allows objects to be staged from tape to disk for faster read access.
  • Versioning – BlackPearl will allow multiple versions of the same objects to be uploaded and saved

In order to implement these new features, changes to the BlackPearl API are required that could affect existing BlackPearl clients. With some major BlackPearl releases (3.0, 5.0, etc.), changes to the BlackPearl API are introduced to support new features and to remove features that have been deprecated. We do not take these API changes lightly, as we know that any change can affect the functionality of BlackPearl clients. We do everything we can to limit these changes while still allowing for the addition of new features.

The changes in 5.0 will require all BlackPearl clients to be updated, recertified, and retested before they can be used with BlackPearl 5.0. At a high level, here are some of the changes being made to the API that could affect existing clients:

Syntactic changes:

  • Removed fields in payload responses -- can cause errors when these entities are queried.
    • Tapes and pools no longer have storage domain ids -- they have storage domain member ids
    • Objects no longer have the old “version” field because the id is now considered the version
    • Data Policy no longer has “always replicate deletes” or “LTFS naming allowed” -- this functionality has been moved
  • Removed request parameters -- can cause errors if a customer tries to use them
    • The “import conflict resolution mode” query parameter for performing tape and pool imports has been removed
    • The following bucket attributes have been removed:
    • Items in the previous section can no longer be filtered by their removed attributes. For example, “get tapes” cannot be filtered by “storage domain id”
  • XML payload attributes that are no longer valid:
    •   priority (now specified as a query parameter)
    •   chunk_client_processing_order_guarantee (now specified as a query parameter)
    •   write optimization (deprecated for xml payloads)
  • Changes to enums -- this includes things like new error message types that can cause errors when received by older clients
    • “Keep multiple versions” is a new versioning type.
    • “Auto compaction in progress” is a new tape type

Semantic Changes:

  • Requests that used to return a 503 “object not uploaded yet” now return a 404.
  • Objects not fully uploaded are no longer listed in an AWS-style “get bucket” command
  • Objects are now marked “latest” once they fully arrive in cache -- not when the job is created.
  • CacheAvailableRetryAfterInSeconds decreased from 300 seconds to 60 seconds -- this is how long Black Pearl recommends the client wait between checking for work to be done on a job
  • The “import conflict resolution mode” query parameter for performing tape and pool imports is still available as an optional parameter, but is ignored if sent.

You can view the full API difference comparison between BlackPearl 4.1 and 5.0. Note that the BlackPearl 5.0 API is still subject to change.

At the very least, clients will need to update to a new release of our SDKs, which will be available on our Downloads page. In some cases, code changes may be required to client integrations.

We have released the 5.0 beta SDKs for Java and .NET now. The Python, C, and Golang SDKs for 5.0 should be available shortly. We also have a BlackPearl running the 5.0 beta release that is available over the Internet. Please Contact the Developer Program Team when you wish to access it.

Testing and recertification will be required to ensure that your client works with BlackPearl 5.0. We will require you to redo the file transfer tests (tests 5 through 8) specified in our Certification Program Test Plan. These tests will need to be done in our lab or at a facility with a BlackPearl and tape library, and the tests results need to be approved by Spectra Logic in order to receive certification for BlackPearl 5.

Please Contact the Developer Program Team if you require assistance.

New BlackPearl 4.0 Simulator – Includes Simulated Tape Storage

We have released a new version of our BlackPearl simulator, version 4.0, to go with our BlackPearl 4.0 software release this week. In addition to having the latest BlackPearl code, this simulator now also includes a simulated tape library. Previous versions of the simulator only included storage of files in the BlackPearl disk cache. This new version allows files to be stored to cache and then to tape, providing a more realistic version of how a typical BlackPearl behaves when connected to a tape library. This also means that BlackPearl archive operations -- called “Jobs” -- can now reach a final completed state.

BlackPearl client developers, partners, and customers should find this new version of the simulator much more useful for their BlackPearl testing. Please Contact the BlackPearl Developer Program if you have any questions or feedback.

BlackPearl SDK for Python 3 Now Available

Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We provide these SDKs in Java, C#/.NET, C, and Python. Our initial Python SDK was built to be used with Python 2. We have now also released a Python SDK that is intended for Python 3.

You can download the Python 3 SDK now. Note that this is a beta release, so please send us any feedback you have on this SDK.

If you are not sure which Python version to use, read about Python 2 versus Python 3 on Python.org.

Importing Foreign LTFS Tapes into BlackPearl

When BlackPearl writes data to tape, it uses the open Linear Tape File System (LTFS) file format. Because of this LTFS support, Spectra Logic was able to add the ability to import non-BlackPearl or “foreign” LTFS tapes to BlackPearl. This is useful for any customer that receives LTFS-formatted tapes from another source and wishes to read those same tapes in the BlackPearl environment. This workflow is particularly common in the Media and Entertainment industry as a way to transfer video files from one group to another. Since every application may utilize the open LTFS format in a different manner when writing data to tapes, it is important that the user verify the various LTFS tape formats will be properly imported for read only use in the BlackPearl environment.

BlackPearl will support the import of foreign LTFS tapes in version 3.5, which is due out in Q1 of 2017. Importing can be done manually via the BlackPearl web management interface or via an external application that calls the BlackPearl API. The “import” process we discuss herein assumes that the foreign LTFS tapes have already been physically imported into the tape library partition to which BlackPearl is connected. Additionally, when a foreign LTFS tape is added to a BlackPearl tape partition, the write-protect switch on the tape cartridge must be set to “read only” before BlackPearl will allow the tape to be imported.  We strongly recommend that the tape be kept in read-only mode so that BlackPearl will not be able to modify the tape in any way. Also note that the API import commands must be initiated by the administrator or “spectra” user.

To manually import foreign LTFS tapes via the web management interface of BlackPearl, users will go to the Tape Management page (Status > Tape Management), click on the tape(s) to be imported, and then go to the Action menu and select Import Foreign Tape.

The tapes can also be imported via an external application that calls the BlackPearl API. While there are several possible workflows that can be taken to import the tapes into BlackPearl by an application, this is one we are currently recommending to our partners:

  1. Application calls API command Get Tapes to get a list of all tapes and their state. Tapes with a state of LTFS_WITH_FOREIGN_DATA should be recorded along with their barcodes.
  2. Application calls API command Raw Import All Tapes to start the process of importing the foreign LTFS tapes. Make sure to make this call using the admin user “spectra”. The application must provide a bucket name in which to import the files on the foreign LTFS tapes. Alternatively the application can call Raw Import Tape to import individual foreign LTFS tapes based on the barcodes recorded from Step 1 above. Again, each time this is called, a bucket name will need to be provided and using this method can allow the application to import different tapes into different buckets if desired.
  3. Application should periodically call API command Get Tapes again to check the state of all tapes. Once there are no more tapes with a state of LTFS Foreign, it means all foreign LTFS tapes have been successfully imported.
  4. Application calls API command Get Physical Placement for Object Parts on Tape to get a list of all files that are on each LTFS tape, using the barcodes recorded in Step 1 above as the input parameter for this call. The application can then read each of these files if needed, such as to create video proxies or to copy them to another location in BlackPearl. If the application doesn’t need to read the files, it can simply record the list of file names.
  5. If the application no longer needs the files on the foreign LTFS tapes, for example because it has read the files and copied them to another bucket in BlackPearl, it can then issue an Eject Tape command for each tape.

As you can see, it is quite easy for applications to work with foreign LTFS tapes in BlackPearl. It is important to remember that the user and/or developer need to confirm that the user’s LTFS tapes can be read by BlackPearl by testing them out. Also, note that BlackPearl will not generate a checksum for the files on foreign LTFS tapes as it does for other files that are imported into BlackPearl. We also don’t currently support importing foreign LTFS tapes that use “tape spanning”, or spanning one file across multiple tapes.

Foreign LTFS tape import is supported in released Java SDK 3.4.0-RC1, and it will be supported by any BlackPearl hardware with code 3.5 and higher. We will soon (in the next two months) also have the foreign LTFS tape import operations available in our other SDKs (C#/.NET, Python, C). We will also soon update our BlackPearl API documentation to include the new foreign LTFS tape import operations.

If you want help working with foreign LTFS tapes in your BlackPearl integration, please contact the Developer Program.

Developer Summit 2016 Recording and Slides

On November 10, 2016, we had a successful, second annual BlackPearl Developer Summit. We have provided a recording, slides, and agenda from the Summit below.

Summit Recording



  • Corporate updates from Spectra CEO Nathan Thompson
  • BlackPearl product overview and enhancements
  • Learn what tools are available to help develop a BlackPearl client
  • Partner presentation detailing client development
  • Learn how easy it is to integrate an existing BlackPearl client into your workflow
  • Demonstration of Avid PAM and BlackPearl integration
  • Demonstration of CatDV BlackPearl integration
  • Question and answer with our BlackPearl Developer Program Team



Second Annual BlackPearl Developer Summit – Nov 10 2016

Thursday, November 10, 2016 — 9:00 a.m. MDT (UTC -7), WebEx

Join us for Spectra Logic’s second annual BlackPearl Developer Summit, a virtual conference for current and potential developers of Spectra® BlackPearl® Deep Storage Gateway.

Event Agenda:

  • Corporate updates from Spectra CEO Nathan Thompson
  • BlackPearl product overview and enhancements
  • Learn what tools are available to help develop a BlackPearl client
  • Partner presentation detailing client development
  • Learn how easy it is to integrate an existing BlackPearl client into your workflow
  • Demonstration of Avid PAM and BlackPearl integration
  • Question and answer with our BlackPearl Developer Program Team

November 10, 2016 — 9:00 a.m. MDT (UTC -7)
Check Time by Country