Second Annual BlackPearl Developer Summit – Nov 10 2016

Thursday, November 10, 2016 — 9:00 a.m. MDT (UTC -7), WebEx

Join us for Spectra Logic’s second annual BlackPearl Developer Summit, a virtual conference for current and potential developers of Spectra® BlackPearl® Deep Storage Gateway.

Event Agenda:

  • Corporate updates from Spectra CEO Nathan Thompson
  • BlackPearl product overview and enhancements
  • Learn what tools are available to help develop a BlackPearl client
  • Partner presentation detailing client development
  • Learn how easy it is to integrate an existing BlackPearl client into your workflow
  • Demonstration of Avid PAM and BlackPearl integration
  • Question and answer with our BlackPearl Developer Program Team

November 10, 2016 — 9:00 a.m. MDT (UTC -7)
Check Time by Country

REGISTER NOW


Python SDK 3.0 Released

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for the Python programming language. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the Python SDK (1.2.0) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

Significant code structure changes were made between the 1.2.0 and 3.0.0 release of the Python SDKs. If you will be upgrading an existing client from 1.2.0 to 3.0.0, you should read our Migration Guide, which describes the changes in more detail. We do not anticipate major code structure changes in the future.

The previous 1.x versions of the Python SDK required that that C SDK also be installed. This new 3.0.0 Python SDK is no longer dependent on the C SDK and therefore no longer requires that the C SDK be installed.

We call this 3.0.0 release a “Pre-Release” because, while it has been tested internally, it is not yet in use by any of our partners or customers.

With this SDK release, all four of our SDKs -- Python, C, Java, and .NET/C# -- are now up to date with BlackPearl 3.x code and support all BlackPearl API commands.

If you have any questions or run into any problems, please post your questions to our Forum.


BlackPearl 3.0 C SDK Now Available

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for the C programming language. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the C SDK (1.2.0) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

Some code structure changes were made between the 1.2.0 and 3.0.0 release of the C SDKs. If you will be upgrading an existing client from 1.2.0 to 3.0.0, you should read our previous blog post Preparing for the BlackPearl 3.0 Release, which describes the changes in more detail.

We call this 3.0.0 release a “Pre-Release” because, while it has been tested internally, it is not yet in use by any of our partners or customers.

If you have any questions or run into any problems, please post your questions to our Forum.


Managing Ejected Tapes in BlackPearl

BlackPearl is an object storage gateway to Spectra’s archive disk and tape library products. Tape libraries have a unique feature not typically available in disk systems – the tapes can be ejected and removed from the tape library. This removal can allow for additional data protection by taking the tapes offsite, and it can also facilitate sharing of the data via physical transport. But once ejected and removed, the data on the tape can no longer be accessed by the tape library.

When a BlackPearl client requests data to be restored from BlackPearl, it is possible that the only copy of the data resides on tape(s) that have been ejected from the library. BlackPearl clients should be designed to handle this ejected tape scenario. Handling the ejected tape scenario is a requirement to pass Spectra certification.

BlackPearl Response to Data on Ejected Tape

When a client attempts to restore data from BlackPearl, it first issues a Create Bulk Get API command to BlackPearl. This command is used by the client to let BlackPearl know which files it wants to retrieve. If any of these files are only available on ejected tape(s), then BlackPearl’s raw API response will be as follows:

If the BlackPearl client is using one of our Software Development Kits (SDKs), which themselves call the Create Bulk Get API command, the SDK will throw an exception in this scenario. Here is the exception thrown by the Java SDK when requested files are only on ejected tapes:

Properly Handling Requests for Data on Ejected Tapes

You can see that in both responses above, BlackPearl indicates that one or more needed tapes are outside of the tape library and provides the barcode label(s) of the tape(s) that have been ejected in brackets. In the example above, the barcode label of the ejected tape is TEST079L7 . In most cases, simply displaying this message to the user should be adequate for the user to use the barcode label to go find the tapes and place them back in the tape library. The user can then try their restore request again and BlackPearl should be able to service it.

In some cases, the user may need more information about the tapes than just the barcode label. Tapes also have other ejection information associated with them, such as Eject Label and Eject Location, that users may need to find the tape. In that case, clients can register to receive a job failure notification. A job is what BlackPearl attempts to create when the Create Bulk Get command above is called. A notification is a message that BlackPearl sends to the client application. If the Create Bulk Get command can’t be serviced because a needed tape is outside of the library, then the job creation fails. BlackPearl clients can register to receive Job Creation Failed Notifications. These notifications includes detailed information about the ejected tapes that are needed to service the file restore request, including the Eject Label and Eject Location.

If a BlackPearl client wants to take a more proactive approach, the client can provide users with a means of checking to see if tapes are ejected before attempting a file restore. The Get Physical Placement API call, and its associated SDK calls, can be used to check the placement of a file or list of files. The Get Physical Placement response will include a list of specific locations where the files are located, whether that location is on disk, tape, or another BlackPearl. If the files are located on tape, the response will include the tape barcode labels and whether or not the tapes are ejected from the library. If the tapes are ejected, additional ejection information will be provided, including the Eject Label and Eject Location.

Message from BlackPearl Web Interface About Ejected Tapes

The BlackPearl web management interface includes a Messages page (Status > Messages) for viewing system event messages. Users can also set up their BlackPearl user account to receive these messages via email.

If a BlackPearl client attempts to restore files that are only available on ejected tapes, a warning message will be added to the Messages page. The message will indicate that a GET request was attempted but couldn’t be serviced because it requires tapes that are ejected from the library. The message will also include the barcode label, Eject Label, and Eject Location of all required ejected tapes. While these messages can be useful to the user, BlackPearl clients shouldn’t depend on the user having to access this Messages page to learn about the ejected tapes.

Conclusion

Developers should ensure that their BlackPearl client applications properly handle requests for data on ejected tapes. There are several methods available to address this scenario. Please Contact the Developer Team if you have any questions about managing ejected tapes.

 


New BlackPearl 3.0 C#/.NET SDK Now Available

We have released new version 3.0.0 of our BlackPearl Software Development Kit (SDK) for C#/.NET. It is now available for download on GitHub. You can also view code documentation, code examples, and installation instructions on our Documentation page.

This new SDK is compatible with our current BlackPearl 3.0.1 release and allows access to all BlackPearl 3.0 API commands. Note that the previous version of the C#/.NET SDK (1.2.5) is also compatible with the current BlackPearl 3.0.1 release but does not give access to all API commands.

We call this 3.0.0 release a “Release Candidate” (RC) because, while it has been tested internally, it is not yet in use by any of our partners or customers.

If you have any questions or run into any problems, please post your questions to our Forum.


Global Bucket Access Control Lists

We discussed BlackPearl bucket-level Access Control Lists (ACLs) in a past blog post. Bucket-level ACLs define the permissions on individual buckets. Global Bucket ACLs are also part of BlackPearl’s ACL framework and define what actions a user or group can take on all buckets stored on the BlackPearl. The Global Bucket ACL is independent of the bucket-level ACLs and bucket owner.

The following Global Bucket ACL permissions can be granted to users and groups:

  • List -- The user can see all buckets and can list the objects in all buckets.
  • Read -- The user can get objects and create GET jobs in all buckets.
  • Write -- The user can put objects and create PUT jobs in all buckets.
  • Delete -- The user can delete objects in all buckets, but cannot delete buckets.
  • Job -- The user can modify or cancel jobs in all buckets created by other users. The user can also see the details of jobs created by other users. Note that all users can view all jobs, but by default, only the initiator of the job can see the full details of a job.
  • Owner -- The user receives full access to all buckets, including all permissions listed above.

How to Use Global Bucket ACLs

Allowing certain users to access all the buckets on a BlackPearl system might be a key administration requirement at some BlackPearl sites.

In the below example the user has been granted global “List” rights. This means when this user logs in not only can they list every bucket on the system they can also list the contents of every bucket.

2016GlobalACLBlogPost-1

The Java Command Line Interface (CLI) is a simple BlackPearl client can be used to view these settings in action. The Java CLI get_service command will provide a list of all buckets on the BlackPearl to which a user has access. Because the user account being used by the Java CLI has global “List” rights, all buckets on the BlackPearl are displayed as shown below.

2016GlobalACLBlogPost-2

The “List” rights also allow the user to get a list of objects in a bucket, in this case the “Jom” bucket. The results are shown below.

2016GlobalACLBlogPost-3

The user does not have rights to retrieve/GET objects from any buckets. As shown in the image below, if the user attempts to get a file, they receive an error.

2016GlobalACLBlogPost-4

We could give the user the ability to retrieve/GET objects from BlackPearl by checking the “Read” permission box as shown in the image below.

2016GlobalACLBlogPost-5

When to Use Global Bucket ACLs

The main reason to use Global Bucket ACLs is for ease of administration. Use of Global Bucket ACLs could allow a user or set of users to administer all the buckets without having to log in with the primary administrator “Spectra” user account. One way to do that would be to set up a group of users and to apply the Global Bucket ACLs to that group. That way there is no need to edit individual user accounts to change access levels. You just simply add or remove users from the group. You can see an example below in the group settings dialog.

2016GlobalACLBlogPost-6

By using the Global Bucket ACLs, users are being granted access not only to existing buckets, but also buckets that will be created in the future. This may save the administrator the time of having to grant access to individual buckets as the buckets are created.


BlackPearl Certification Program

Spectra Logic has created the BlackPearl Certification Program to ensure that the BlackPearl clients created by our partners are robust and reliable. We have provided detailed information on how to get certified. All partners should strive to have their BlackPearl clients certified. We are now certifying our first clients and will be announcing them soon. If you are interested in having your client certified, please Contact Us.

BlackPearl Certification Program


Change to Package Structure in Java 3.x RC SDK

The Java 3.0 and 3.2 Release Candidate (RC) SDKs are available and some partners and customers are now trying them out. Yesterday we had to make a change to the package structure in these SDKs to make it very clear which classes are auto generated and which are not by moving the classes that are not auto generated to different packages. You can view the details in this GitHub pull request. Most users probably aren’t yet using these classes, but if they are, they will need to update their code to support the new package names. Contact us on the forums if you have any questions or concerns.


Developer Program Team at NAB 2016

The biggest trade show in the Media and Entertainment arena, NAB 2016, is coming next month to Las Vegas, and Spectra Logic will be there in full force. The Spectra Logic Developer Program team will also be on site to answer your questions and discuss your BlackPearl client integration plans. Included in our team will be Ryan Moore, BlackPearl SDK Engineering Manager, and yours truly, BlackPearl Developer Evangelist Jeff Braunstein. Come visit us at booth SL11816 and tell us about your plans to integrate with BlackPearl.


New Article About Exceptions and Unexpected Conditions

Any software application has to deal with errors, exceptions, and unexpected conditions. BlackPearl client developers must consider what unexpected conditions and exceptions may occur in their clients and have the client respond to them appropriately. We have created a new article that reviews the most common unexpected conditions in a BlackPearl client and how to respond to them appropriately. We recommend all developers read it and ensure that their clients follow these guidelines.

Managing Unexpected Conditions in BlackPearl Applications