We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for the C programming language. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the C SDK (1.2.0) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

Some code structure changes were made between the 1.2.0 and 3.0.0 release of the C SDKs. If you will be upgrading an existing client from 1.2.0 to 3.0.0, you should read our previous blog post Preparing for the BlackPearl 3.0 Release, which describes the changes in more detail.

We call this 3.0.0 release a “Pre-Release” because, while it has been tested internally, it is not yet in use by any of our partners or customers.

If you have any questions or run into any problems, please post your questions to our Forum.

BlackPearl is an object storage gateway to Spectra’s archive disk and tape library products. Tape libraries have a unique feature not typically available in disk systems – the tapes can be ejected and removed from the tape library. This removal can allow for additional data protection by taking the tapes offsite, and it can also facilitate sharing of the data via physical transport. But once ejected and removed, the data on the tape can no longer be accessed by the tape library.

When a BlackPearl client requests data to be restored from BlackPearl, it is possible that the only copy of the data resides on tape(s) that have been ejected from the library. BlackPearl clients should be designed to handle this ejected tape scenario. Handling the ejected tape scenario is a requirement to pass Spectra certification.

BlackPearl Response to Data on Ejected Tape

When a client attempts to restore data from BlackPearl, it first issues a Create Bulk Get API command to BlackPearl. This command is used by the client to let BlackPearl know which files it wants to retrieve. If any of these files are only available on ejected tape(s), then BlackPearl’s raw API response will be as follows:

If the BlackPearl client is using one of our Software Development Kits (SDKs), which themselves call the Create Bulk Get API command, the SDK will throw an exception in this scenario. Here is the exception thrown by the Java SDK when requested files are only on ejected tapes:

Properly Handling Requests for Data on Ejected Tapes

You can see that in both responses above, BlackPearl indicates that one or more needed tapes are outside of the tape library and provides the barcode label(s) of the tape(s) that have been ejected in brackets. In the example above, the barcode label of the ejected tape is TEST079L7 . In most cases, simply displaying this message to the user should be adequate for the user to use the barcode label to go find the tapes and place them back in the tape library. The user can then try their restore request again and BlackPearl should be able to service it.

In some cases, the user may need more information about the tapes than just the barcode label. Tapes also have other ejection information associated with them, such as Eject Label and Eject Location, that users may need to find the tape. In that case, clients can register to receive a job failure notification. A job is what BlackPearl attempts to create when the Create Bulk Get command above is called. A notification is a message that BlackPearl sends to the client application. If the Create Bulk Get command can’t be serviced because a needed tape is outside of the library, then the job creation fails. BlackPearl clients can register to receive Job Creation Failed Notifications. These notifications includes detailed information about the ejected tapes that are needed to service the file restore request, including the Eject Label and Eject Location.

If a BlackPearl client wants to take a more proactive approach, the client can provide users with a means of checking to see if tapes are ejected before attempting a file restore. The Get Physical Placement API call, and its associated SDK calls, can be used to check the placement of a file or list of files. The Get Physical Placement response will include a list of specific locations where the files are located, whether that location is on disk, tape, or another BlackPearl. If the files are located on tape, the response will include the tape barcode labels and whether or not the tapes are ejected from the library. If the tapes are ejected, additional ejection information will be provided, including the Eject Label and Eject Location.

Message from BlackPearl Web Interface About Ejected Tapes

The BlackPearl web management interface includes a Messages page (Status > Messages) for viewing system event messages. Users can also set up their BlackPearl user account to receive these messages via email.

If a BlackPearl client attempts to restore files that are only available on ejected tapes, a warning message will be added to the Messages page. The message will indicate that a GET request was attempted but couldn’t be serviced because it requires tapes that are ejected from the library. The message will also include the barcode label, Eject Label, and Eject Location of all required ejected tapes. While these messages can be useful to the user, BlackPearl clients shouldn’t depend on the user having to access this Messages page to learn about the ejected tapes.

Conclusion

Developers should ensure that their BlackPearl client applications properly handle requests for data on ejected tapes. There are several methods available to address this scenario. Please Contact the Developer Team if you have any questions about managing ejected tapes.

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for C#/.NET. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the C#/.NET SDK (1.2.5) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

We call this 3.0.0 release a “Release Candidate” (RC) because, while it has been tested internally, it is not yet in use by any of our partners or customers.

If you have any questions or run into any problems, please post your questions to our Forum.

We discussed BlackPearl bucket-level Access Control Lists (ACLs) in a past blog post. Bucket-level ACLs define the permissions on individual buckets. Global Bucket ACLs are also part of BlackPearl’s ACL framework and define what actions a user or group can take on all buckets stored on the BlackPearl. The Global Bucket ACL is independent of the bucket-level ACLs and bucket owner.

The following Global Bucket ACL permissions can be granted to users and groups:

• List -- The user can see all buckets and can list the objects in all buckets.
• Read -- The user can get objects and create GET jobs in all buckets.
• Write -- The user can put objects and create PUT jobs in all buckets.
• Delete -- The user can delete objects in all buckets, but cannot delete buckets.
• Job -- The user can modify or cancel jobs in all buckets created by other users. The user can also see the details of jobs created by other users. Note that all users can view all jobs, but by default, only the initiator of the job can see the full details of a job.
• Owner -- The user receives full access to all buckets, including all permissions listed above.

How to Use Global Bucket ACLs

Allowing certain users to access all the buckets on a BlackPearl system might be a key administration requirement at some BlackPearl sites.

In the below example the user has been granted global “List” rights. This means when this user logs in not only can they list every bucket on the system they can also list the contents of every bucket.

The Java Command Line Interface (CLI) is a simple BlackPearl client can be used to view these settings in action. The Java CLI get_service command will provide a list of all buckets on the BlackPearl to which a user has access. Because the user account being used by the Java CLI has global “List” rights, all buckets on the BlackPearl are displayed as shown below.

The “List” rights also allow the user to get a list of objects in a bucket, in this case the “Jom” bucket. The results are shown below.

The user does not have rights to retrieve/GET objects from any buckets. As shown in the image below, if the user attempts to get a file, they receive an error.

We could give the user the ability to retrieve/GET objects from BlackPearl by checking the “Read” permission box as shown in the image below.

When to Use Global Bucket ACLs

The main reason to use Global Bucket ACLs is for ease of administration. Use of Global Bucket ACLs could allow a user or set of users to administer all the buckets without having to log in with the primary administrator “Spectra” user account. One way to do that would be to set up a group of users and to apply the Global Bucket ACLs to that group. That way there is no need to edit individual user accounts to change access levels. You just simply add or remove users from the group. You can see an example below in the group settings dialog.

By using the Global Bucket ACLs, users are being granted access not only to existing buckets, but also buckets that will be created in the future. This may save the administrator the time of having to grant access to individual buckets as the buckets are created.

Spectra Logic has created the BlackPearl Certification Program to ensure that the BlackPearl clients created by our partners are robust and reliable. We have provided detailed information on how to get certified. All partners should strive to have their BlackPearl clients certified. We are now certifying our first clients and will be announcing them soon. If you are interested in having your client certified, please Contact Us.

BlackPearl Certification Program

The Java 3.0 and 3.2 Release Candidate (RC) SDKs are available and some partners and customers are now trying them out. Yesterday we had to make a change to the package structure in these SDKs to make it very clear which classes are auto generated and which are not by moving the classes that are not auto generated to different packages. You can view the details in this GitHub pull request. Most users probably aren’t yet using these classes, but if they are, they will need to update their code to support the new package names. Contact us on the forums if you have any questions or concerns.

The biggest trade show in the Media and Entertainment arena, NAB 2016, is coming next month to Las Vegas, and Spectra Logic will be there in full force. The Spectra Logic Developer Program team will also be on site to answer your questions and discuss your BlackPearl client integration plans. Included in our team will be Ryan Moore, BlackPearl SDK Engineering Manager, and yours truly, BlackPearl Developer Evangelist Jeff Braunstein. Come visit us at booth SL11816 and tell us about your plans to integrate with BlackPearl.

Any software application has to deal with errors, exceptions, and unexpected conditions. BlackPearl client developers must consider what unexpected conditions and exceptions may occur in their clients and have the client respond to them appropriately. We have created a new article that reviews the most common unexpected conditions in a BlackPearl client and how to respond to them appropriately. We recommend all developers read it and ensure that their clients follow these guidelines.

Managing Unexpected Conditions in BlackPearl Applications

We are happy to announce that Version 3.0 of BlackPearl has been released! It is now shipping on new BlackPearl hardware, and it is available to existing BlackPearl customers. Existing BlackPearl customers should work with Spectra Logic Support to ensure that the upgrade goes smoothly.

We explained the main new features of BlackPearl in two recent blog posts:

• BlackPearl 3.0 New Features Part 1: ArcticBlue and Advanced Bucket Management
  
• BlackPearl 3.0 New Features Part 2: Access Control Lists

To assist developers with building BlackPearl 3.0 clients, we have released the following new tools and resources:

• 3.0 Simulator -- simulates a working BlackPearl for easy development
• 3.0 API Reference -- review the operations supported by BlackPearl’s API
• 3.0 Java SDK -- Use the new SDK to take advantage of all the new BlackPearl features
• 3.0 Java SDK Javadocs -- Learn about all the SDK classes, methods, properties, and events

Over the coming weeks we will be releasing 3.0 versions of our SDKs in our remaining languages -- .NET/C#, C, and Python. Note that all of the existing 1.x SDKs work fine with BlackPearl 3.0, they just don’t yet have access to the new 3.0 features.

We will be releasing more resources and information over time to help you more easily build BlackPearl 3.0 clients.

As with most cloud storage systems, when a file is uploaded to BlackPearl we call it an “object”. There are some constraints around how objects are named in BlackPearl due to it often being connected to a tape system that stores data using the open LTFS format.

When you configure BlackPearl, you set up Storage Domain(s), which are collections of tape and/or disk partitions. If the Storage Domain includes tape partition(s), you must specify the “LTFS File Name” option for the Storage Domain. This option specifies how the file is named when it is placed on tape. There are two options for the LTFS File Name:

• Object Name — LTFS file names use the format {bucket name}/{object name}, for example bucket1/video1.mov. Object names must comply with LTFS 2.4 specification file naming rules. The file name must be 255 characters or less. If the tapes are ejected from the BlackPearl gateway and loaded into a non‐BlackPearl tape partition, the file names match the object names. The colon character (:) is not allowed in LTFS file names and therefore not allowed in BlackPearl object names. The slash character (/) is also technically not allowed in LTFS files names; however, in BlackPearl a slash is allowed in the object name and will get translated as a directory in the LTFS file system (e.g. directory1/directory2/video1.mov). The following characters are not recommended in LTFS file names or BlackPearl object names for reasons of cross-platform compatibility: control characters such as carriage return (CR) and line feed (LF), double quotation mark (“), asterisk (*), question mark (?), less than sign (<), greater than sign (>),  backslash (\), vertical line (|). Note that per the LTFS specification, “Implementations which claim compliance with version 2.4.0 or later of this specification shall support the percent-encoding of names … in order to avoid issues with the characters listed [above].”
• Object ID — LTFS file names use the format {bucket name}/{object id}, for example bucket1/1fc6f09c‐dd72‐41ea‐8043‐0491ab8a6d82. Object names do not need to comply with LTFS file naming rules. The object names are saved as LTFS extended attributes allowing any third party application to reconstruct all the data including the object names.

When building a BlackPearl client, developers should be prepared for customers to choose either LTFS File Name option, and thus should ensure that object names are LTFS compliant as described in the first option above.