To date, hundreds of Spectra Logic customers have implemented our BlackPearl® Converged Storage System to affordably store data long-term. Spectra’s BlackPearl is a purpose-built storage platform that integrates directly with ”data mover” software applications to simplify workflows and seamlessly manage large volumes of data to multiple storage targets. BlackPearl includes an S3-like interface to move data into its object storage on the cloud, disk, and tape. In order to help even more customers meet their long-term storage needs, Spectra has recently introduced another, easier system for transferring files into BlackPearl. This new system is called Spectra RioBroker. The announcement, made yesterday, can be read here.

As a software front-end to BlackPearl, Spectra RioBroker acts as a data mover for applications that wish to move data into the BlackPearl object storage gateway. The typical system architecture is shown in the diagram below.

There are several advantages to using Spectra RioBroker instead of the direct BlackPearl i­nterface:

• Enables easier client development for partners with a simple abstraction layer over BlackPearl interface
• Allows more clients and applications to share BlackPearl object storage resources in parallel at ever higher performance
• Provides remote input/output capabilities to multiple Spectra BlackPearl Converged Storage servers
• Brokers data stream input and output between multiple sources and destinations

• Provides for enterprise-level high availability*
• Delivers ultra-high performance with clustering capabilities*
• Facilitates seamless content migration from legacy storage software to a modern solution

A perfect use case example for Spectra RioBroker is for Media Asset Management (MAM) software typically used in the Media and Entertainment industry. Spectra’s MAM partners have built integrations to Spectra RioBroker so that their customers can archive and restore files directly from the MAM interface without the files passing through the MAM server. Now that the software is released, we expect many more partners and customers to build integrations.

The Spectra RioBroker software application is provided to Spectra customers at no cost. The software is currently available for Windows servers, and a Linux version will be released in the future. Both partners and customers can utilize the free tools on Spectra’s Developer Program Website to build their own integrations. The website includes all the tools needed to build an integration, including the Spectra RioBroker installer, a BlackPearl simulator, an SDK, code examples, documentation and more. Spectra RioBroker, like BlackPearl, uses a RESTful API with very simple commands. A basic integration to Rio Broker requires only three API commands – Archive, Restore, and Check Job Status.

While Spectra RioBroker will become a popular method to integrate to BlackPearl, direct integration to BlackPearl will continue to be logical for certain use cases. Spectra Logic will continue to support this direct integration method.

*Available in future release of Spectra RioBroker

We have released a new BlackPearl simulator which is running the latest version of our BlackPearl 5.0 code, scheduled for release in late June. You can get the simulator on our Downloads page. SDKs for BlackPearl 5 are also available on this page. Make sure to carefully follow the simulator setup instructions. Contact the Developer Program Team if you need assistance.

When BlackPearl writes data to tape, it uses the open Linear Tape File System (LTFS) file format. Because of this LTFS support, Spectra Logic was able to add the ability to import non-BlackPearl or “foreign” LTFS tapes to BlackPearl. This is useful for any customer that receives LTFS-formatted tapes from another source and wishes to read those same tapes in the BlackPearl environment. This workflow is particularly common in the Media and Entertainment industry as a way to transfer video files from one group to another. Since every application may utilize the open LTFS format in a different manner when writing data to tapes, it is important that the user verify the various LTFS tape formats will be properly imported for read only use in the BlackPearl environment.

BlackPearl will support the import of foreign LTFS tapes in version 3.5, which is due out in Q1 of 2017. Importing can be done manually via the BlackPearl web management interface or via an external application that calls the BlackPearl API. The “import” process we discuss herein assumes that the foreign LTFS tapes have already been physically imported into the tape library partition to which BlackPearl is connected. Additionally, when a foreign LTFS tape is added to a BlackPearl tape partition, the write-protect switch on the tape cartridge must be set to “read only” before BlackPearl will allow the tape to be imported.  We strongly recommend that the tape be kept in read-only mode so that BlackPearl will not be able to modify the tape in any way. Also note that the API import commands must be initiated by the administrator or “spectra” user.

To manually import foreign LTFS tapes via the web management interface of BlackPearl, users will go to the Tape Management page (Status > Tape Management), click on the tape(s) to be imported, and then go to the Action menu and select Import Foreign Tape.

The tapes can also be imported via an external application that calls the BlackPearl API. While there are several possible workflows that can be taken to import the tapes into BlackPearl by an application, this is one we are currently recommending to our partners:

1. Application calls API command Get Tapes to get a list of all tapes and their state. Tapes with a state of LTFS_WITH_FOREIGN_DATA should be recorded along with their barcodes.
2. Application calls API command Raw Import All Tapes to start the process of importing the foreign LTFS tapes. Make sure to make this call using the admin user “spectra”. The application must provide a bucket name in which to import the files on the foreign LTFS tapes. Alternatively the application can call Raw Import Tape to import individual foreign LTFS tapes based on the barcodes recorded from Step 1 above. Again, each time this is called, a bucket name will need to be provided and using this method can allow the application to import different tapes into different buckets if desired.
3. Application should periodically call API command Get Tapes again to check the state of all tapes. Once there are no more tapes with a state of LTFS Foreign, it means all foreign LTFS tapes have been successfully imported.
4. Application calls API command Get Physical Placement for Object Parts on Tape to get a list of all files that are on each LTFS tape, using the barcodes recorded in Step 1 above as the input parameter for this call. The application can then read each of these files if needed, such as to create video proxies or to copy them to another location in BlackPearl. If the application doesn’t need to read the files, it can simply record the list of file names.
5. If the application no longer needs the files on the foreign LTFS tapes, for example because it has read the files and copied them to another bucket in BlackPearl, it can then issue an Eject Tape command for each tape.

As you can see, it is quite easy for applications to work with foreign LTFS tapes in BlackPearl. It is important to remember that the user and/or developer need to confirm that the user’s LTFS tapes can be read by BlackPearl by testing them out. Also, note that BlackPearl will not generate a checksum for the files on foreign LTFS tapes as it does for other files that are imported into BlackPearl. We also don’t currently support importing foreign LTFS tapes that use “tape spanning”, or spanning one file across multiple tapes.

Foreign LTFS tape import is supported in released Java SDK 3.4.0-RC1, and it will be supported by any BlackPearl hardware with code 3.5 and higher. We will soon (in the next two months) also have the foreign LTFS tape import operations available in our other SDKs (C#/.NET, Python, C). We will also soon update our BlackPearl API documentation to include the new foreign LTFS tape import operations.

If you want help working with foreign LTFS tapes in your BlackPearl integration, please contact the Developer Program.

On November 10, 2016, we had a successful, second annual BlackPearl Developer Summit. We have provided a recording, slides, and agenda from the Summit below.

Summit Recording

Slides

Agenda

• Corporate updates from Spectra CEO Nathan Thompson
• BlackPearl product overview and enhancements
• Learn what tools are available to help develop a BlackPearl client
• Partner presentation detailing client development
• Learn how easy it is to integrate an existing BlackPearl client into your workflow
• Demonstration of Avid PAM and BlackPearl integration
• Demonstration of CatDV BlackPearl integration
• Question and answer with our BlackPearl Developer Program Team

Thursday, November 10, 2016 — 9:00 a.m. MDT (UTC -7), WebEx

Join us for Spectra Logic’s second annual BlackPearl Developer Summit, a virtual conference for current and potential developers of Spectra® BlackPearl® Deep Storage Gateway.

Event Agenda:

• Corporate updates from Spectra CEO Nathan Thompson
• BlackPearl product overview and enhancements
• Learn what tools are available to help develop a BlackPearl client
• Partner presentation detailing client development
• Learn how easy it is to integrate an existing BlackPearl client into your workflow
• Demonstration of Avid PAM and BlackPearl integration
• Question and answer with our BlackPearl Developer Program Team

November 10, 2016 — 9:00 a.m. MDT (UTC -7)
Check Time by Country

REGISTER NOW

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for the Python programming language. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the Python SDK (1.2.0) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

Significant code structure changes were made between the 1.2.0 and 3.0.0 release of the Python SDKs. If you will be upgrading an existing client from 1.2.0 to 3.0.0, you should read our Migration Guide, which describes the changes in more detail. We do not anticipate major code structure changes in the future.

The previous 1.x versions of the Python SDK required that that C SDK also be installed. This new 3.0.0 Python SDK is no longer dependent on the C SDK and therefore no longer requires that the C SDK be installed.

We call this 3.0.0 release a “Pre-Release” because, while it has been tested internally, it is not yet in use by any of our partners or customers.

With this SDK release, all four of our SDKs -- Python, C, Java, and .NET/C# -- are now up to date with BlackPearl 3.x code and support all BlackPearl API commands.

If you have any questions or run into any problems, please post your questions to our Forum.

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for the C programming language. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the C SDK (1.2.0) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

Some code structure changes were made between the 1.2.0 and 3.0.0 release of the C SDKs. If you will be upgrading an existing client from 1.2.0 to 3.0.0, you should read our previous blog post Preparing for the BlackPearl 3.0 Release, which describes the changes in more detail.

We call this 3.0.0 release a “Pre-Release” because, while it has been tested internally, it is not yet in use by any of our partners or customers.

If you have any questions or run into any problems, please post your questions to our Forum.

BlackPearl is an object storage gateway to Spectra’s archive disk and tape library products. Tape libraries have a unique feature not typically available in disk systems – the tapes can be ejected and removed from the tape library. This removal can allow for additional data protection by taking the tapes offsite, and it can also facilitate sharing of the data via physical transport. But once ejected and removed, the data on the tape can no longer be accessed by the tape library.

When a BlackPearl client requests data to be restored from BlackPearl, it is possible that the only copy of the data resides on tape(s) that have been ejected from the library. BlackPearl clients should be designed to handle this ejected tape scenario. Handling the ejected tape scenario is a requirement to pass Spectra certification.

BlackPearl Response to Data on Ejected Tape

When a client attempts to restore data from BlackPearl, it first issues a Create Bulk Get API command to BlackPearl. This command is used by the client to let BlackPearl know which files it wants to retrieve. If any of these files are only available on ejected tape(s), then BlackPearl’s raw API response will be as follows:

If the BlackPearl client is using one of our Software Development Kits (SDKs), which themselves call the Create Bulk Get API command, the SDK will throw an exception in this scenario. Here is the exception thrown by the Java SDK when requested files are only on ejected tapes:

Properly Handling Requests for Data on Ejected Tapes

You can see that in both responses above, BlackPearl indicates that one or more needed tapes are outside of the tape library and provides the barcode label(s) of the tape(s) that have been ejected in brackets. In the example above, the barcode label of the ejected tape is TEST079L7 . In most cases, simply displaying this message to the user should be adequate for the user to use the barcode label to go find the tapes and place them back in the tape library. The user can then try their restore request again and BlackPearl should be able to service it.

In some cases, the user may need more information about the tapes than just the barcode label. Tapes also have other ejection information associated with them, such as Eject Label and Eject Location, that users may need to find the tape. In that case, clients can register to receive a job failure notification. A job is what BlackPearl attempts to create when the Create Bulk Get command above is called. A notification is a message that BlackPearl sends to the client application. If the Create Bulk Get command can’t be serviced because a needed tape is outside of the library, then the job creation fails. BlackPearl clients can register to receive Job Creation Failed Notifications. These notifications includes detailed information about the ejected tapes that are needed to service the file restore request, including the Eject Label and Eject Location.

If a BlackPearl client wants to take a more proactive approach, the client can provide users with a means of checking to see if tapes are ejected before attempting a file restore. The Get Physical Placement API call, and its associated SDK calls, can be used to check the placement of a file or list of files. The Get Physical Placement response will include a list of specific locations where the files are located, whether that location is on disk, tape, or another BlackPearl. If the files are located on tape, the response will include the tape barcode labels and whether or not the tapes are ejected from the library. If the tapes are ejected, additional ejection information will be provided, including the Eject Label and Eject Location.

Message from BlackPearl Web Interface About Ejected Tapes

The BlackPearl web management interface includes a Messages page (Status > Messages) for viewing system event messages. Users can also set up their BlackPearl user account to receive these messages via email.

If a BlackPearl client attempts to restore files that are only available on ejected tapes, a warning message will be added to the Messages page. The message will indicate that a GET request was attempted but couldn’t be serviced because it requires tapes that are ejected from the library. The message will also include the barcode label, Eject Label, and Eject Location of all required ejected tapes. While these messages can be useful to the user, BlackPearl clients shouldn’t depend on the user having to access this Messages page to learn about the ejected tapes.

Conclusion

Developers should ensure that their BlackPearl client applications properly handle requests for data on ejected tapes. There are several methods available to address this scenario. Please Contact the Developer Team if you have any questions about managing ejected tapes.

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for C#/.NET. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the C#/.NET SDK (1.2.5) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

We call this 3.0.0 release a “Release Candidate” (RC) because, while it has been tested internally, it is not yet in use by any of our partners or customers.

If you have any questions or run into any problems, please post your questions to our Forum.

We discussed BlackPearl bucket-level Access Control Lists (ACLs) in a past blog post. Bucket-level ACLs define the permissions on individual buckets. Global Bucket ACLs are also part of BlackPearl’s ACL framework and define what actions a user or group can take on all buckets stored on the BlackPearl. The Global Bucket ACL is independent of the bucket-level ACLs and bucket owner.

The following Global Bucket ACL permissions can be granted to users and groups:

• List -- The user can see all buckets and can list the objects in all buckets.
• Read -- The user can get objects and create GET jobs in all buckets.
• Write -- The user can put objects and create PUT jobs in all buckets.
• Delete -- The user can delete objects in all buckets, but cannot delete buckets.
• Job -- The user can modify or cancel jobs in all buckets created by other users. The user can also see the details of jobs created by other users. Note that all users can view all jobs, but by default, only the initiator of the job can see the full details of a job.
• Owner -- The user receives full access to all buckets, including all permissions listed above.

How to Use Global Bucket ACLs

Allowing certain users to access all the buckets on a BlackPearl system might be a key administration requirement at some BlackPearl sites.

In the below example the user has been granted global “List” rights. This means when this user logs in not only can they list every bucket on the system they can also list the contents of every bucket.

The Java Command Line Interface (CLI) is a simple BlackPearl client can be used to view these settings in action. The Java CLI get_service command will provide a list of all buckets on the BlackPearl to which a user has access. Because the user account being used by the Java CLI has global “List” rights, all buckets on the BlackPearl are displayed as shown below.

The “List” rights also allow the user to get a list of objects in a bucket, in this case the “Jom” bucket. The results are shown below.

The user does not have rights to retrieve/GET objects from any buckets. As shown in the image below, if the user attempts to get a file, they receive an error.

We could give the user the ability to retrieve/GET objects from BlackPearl by checking the “Read” permission box as shown in the image below.

When to Use Global Bucket ACLs

The main reason to use Global Bucket ACLs is for ease of administration. Use of Global Bucket ACLs could allow a user or set of users to administer all the buckets without having to log in with the primary administrator “Spectra” user account. One way to do that would be to set up a group of users and to apply the Global Bucket ACLs to that group. That way there is no need to edit individual user accounts to change access levels. You just simply add or remove users from the group. You can see an example below in the group settings dialog.

By using the Global Bucket ACLs, users are being granted access not only to existing buckets, but also buckets that will be created in the future. This may save the administrator the time of having to grant access to individual buckets as the buckets are created.