How to Use Python 3 BlackPearl Documentation

Spectra BlackPearl includes an Application Program Interface (API) that allows applications to interact with BlackPearl. Spectra Logic has created a number of Software Development Kits (SDKs) in various programming languages, including Python 3, which act as an overlay to the API and make it easier to interact with the API in the selected programming language.

For the Python 3 SDK, documentation has been provided that explains how to use the commands. This documentation is automatically generated from the code. While the documentation is very helpful, it can be difficult to interpret. The example below should assist you in using the documentation.

The first place to start is in the API documentation. You should first review this API documentation and determine which command you wish to use. As an example, perhaps you want to determine the BlackPearl system health. This would be done with the Verify System Health API command. The API documentation for this page includes the command syntax and the request and response parameters, as shown below.

You next need to find the corresponding Python components for this command in the Python documentation. As the API command in the API documentation is titled “Verify System Health”, we now first want to search the Python documentation page for “VerifySystemHealth” (no spaces). You can usually use Ctrl+F or Command+F to search a web page for text. Your search should take you to the part of the page with the associated Python classes for this command, as shown below.

We now need to search the Python documentation page again for the associated function call. Again, as the API command in the API documentation is titled “Verify System Health”, we this time want to search the Python documentation page for “verify_system_health” (all lower case, replace spaces with underscores). We can now see the function name for this API command, as shown below:

Now we will create the Python code to call the Verify System Health command. We first need to make sure that we import the SDK library and that we create a “client” object for the BlackPearl we are calling:

You will need to provide the actual credentials for your BlackPearl system.

Next we will create the Python code for the Verify System Health call:

We can review the various components of the code:

  • getHealthResponse – this is an arbitrary variable name that you choose
  • client – this is the client variable from above when we created the client object
  • verify_system_health_spectra_s3 – this is the function name we found on the Python documentation page
  • ds3.VerifySystemHealthSpectraS3Request() – ds3 is the SDK code library; VerifySystemHealthSpectraS3Request is the class name we found on the Python documentation page
  • DatabaseFilesystemFreeSpace – this is the response parameter we found on the API documentation page
  • MsRequiredToVerifyDataPlannerHealth – this is the response parameter we found on the API documentation page

Hopefully this example will help you use the documentation to create Python scripts and applications for BlackPearl.

RioBroker “Relationship” Feature to Be Removed in 3.0 Release

Spectra RioBroker is a data mover that allows our partners and customers to build solutions that move data into and out of BlackPearl. RioBroker includes a robust API to allow software applications to manage this data movement process.

RioBroker’s API currently includes a “Relationships” feature. The Relationships feature allows a user to specify an ID and link many objects to that ID. An application could then look and find all related files through the relationship ID. Relationships has not seen use by users and was implemented in order to facilitate the linking or aggregating of multiple objects together. In an effort to better solve the problem we are going to remove this feature, and longer term we will provide a solution better tailored to solving the object linking problem.

Starting with the current release of RioBroker, 2.1, Relationships are deprecated, and if a user or integration is using it we recommend you start removing it now. In the following release, 3.0, due in September 2020, the feature will be removed.

For any immediate needs, if having related files being easily identifiable is needed, then we recommend using a common prefix for object names. So if your object name is video.mfx, and has a related file, audio.mxf, then we recommend storing the objects with a name like, uniqueId/video.mfx and uniqueId/audio.mxf. We are currently evaluating how best to link files together into a single logical object for the longer-term solution.

Please Contact the Developer Program if you have any questions or concerns.

No More Releases for Python 2 SDK

The Python Software Foundation has announced the “Sunsetting” of Python 2. Spectra Logic currently provides a BlackPearl Software Development Kit (SDK) using Python 2. With the announced sunsetting of Python 2, Spectra Logic will no longer provide future releases of the Python 2 SDK to match future releases of BlackPearl. This means that any changes or updates to the BlackPearl API will not be available via the Python 2 SDK. The Python 2 SDK should continue to work with future versions of BlackPearl (5.2, 5.3, etc.) until the 6.0 release, at which time there will be changes to the BlackPearl API that will cause the Python 2 SDK to no longer work. There are no plans for a BlackPearl 6.0 release at this time. Whole number changes in the BlackPearl release number (e.g. 5.x, 6.x) indicate breaking API changes.

Customers should work to update to the Python 3 SDK as soon as possible.

Spectra Logic may choose to address bug fixes in the latest Python 2 SDK on a case-by-case basis.

Please contact the Spectra Logic Developer program if you have any questions or concerns.

Restoring Large Files in Windows with BlackPearl – Sparse Files

Spectra Logic provides Software Development Kits (SDKs) for BlackPearl in several different programming languages to make it easier for our partners and customers to create integrations to BlackPearl. Some of the BlackPearl SDKs are used in Windows environments, particularly the Java and .NET SDKs. An issue was recently discovered in the way that these SDKs write large data blocks (64GB+) restored from BlackPearl on to disk.

In Windows, by default, when a file is created, Windows will initially reserve the space for the file on disk by filling the file with zeros. It then later replaces those zeros with the actual file byte content as it is received. When the files are 64GB or larger, it can take a very long time to write all of the zeros. In that time the file transfer from BlackPearl can time out, causing the SDK to have to resubmit its request to BlackPearl to restore the file. This resubmission causes the overall transfer process to take longer than it would if the resubmission did not occur. And in extreme cases with extremely large files (1TB+), the transfer can permanently fail.

To avoid this time out issue, the zeros should not be initially written in the file. This can be done by specifying that Windows Sparse Files be used. With Windows sparse files, per Microsoft, “the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data”. Sparse files are therefore initially written much more quickly than non-sparse files.

We have updated our Java and .NET SDKs to use sparse files on restores. Partners and customers using these SDKs should update to the latest versions to ensure that they have this sparse file fix. However, the fixes in these SDKs will only be helpful if the “FileHelpers” class is being used, in which case the SDKs will be writing the restored files. For those organizations that are using these SDKs but have written their own code to write the restored files, they must implement sparse files themselves in their own code.

You can see examples of how Spectra Logic implemented sparse files in the SDKs:

Please Contact the Spectra Developer Program Team  if you need assistance with implementing sparse files with BlackPearl.

BlackPearl Eon Browser 2.1 Official Release

BlackPearl Eon Browser 2.1 has been officially released. The official release version is 2.1.6. There are many new features and bug fixes since the prior 2.0 release. We recommend users upgrade to 2.1.6 at their earliest convenience. You can get this release on our Downloads page. Contact the Developer Program Team if you have any questions or run into any problems.

Eon Browser 2.1 Beta Available for Testing

We have released a beta of the next version of the Eon Browser, version 2.1.4. You can download the Eon Browser beta on GitHub. A draft version of the new User Guide is also available. Note that this version should not be used in a production environment and will not be supported by Spectra Support. You can submit feedback via GitHub or by Contacting Us. We look forward to receiving your feedback.



Developer Program News – May 2017

The Developer Program provides developers with SDKs, APIs, documentation, and more, to design, create, and develop Spectra BlackPearl Converged Storage Spectra S3 applications with deep storage capabilities. Spectra Logic has seen tremendous success with BlackPearl and has many exciting activities planned in 2017!

Getting Started with BlackPearl Partial File Restore Integration

In the Media & Entertainment world, data files have reached very large sizes, particularly in the case of high resolution video that can exceed 1TB in size. In order to efficiently work with very large files, the media file processing is done in sections, with the end-user requesting content “snippets” based on timecodes. Object storage devices that are used to store very large files are not typically aware of the timecode-to-byte relationship, and do not have the content awareness necessary to extract and create partial media files. To bridge the gap between time and bytes, BlackPearl has added a Partial File Restore (PFR) feature to enable the media processing application to efficiently retrieve a complete media file based on timecode offsets. Read More

BlackPearl SDK for Python 3 Now Available

Spectra Logic provides software development kits (SDKs) to make it easier to create applications that integrate with BlackPearl. We provide these SDKs in Java, C#/.NET, C, and Python. Our initial Python SDK was built to be used with Python 2. We have now also released a Python SDK that is intended for Python 3. Read More

SDK & CLI New Releases; Improvements to BlackPearl 3.0 Compatibility

Our Engineering Team has released new versions of the .NET/C# and Java Software Development Kits (SDK), as well as the Java Command-Line Interface (CLI). There are a number of new features and bug fixes in these releases. Probably the most important new feature in these releases is improved support for the upcoming BlackPearl 3.0 release. We strongly recommend that developers currently building BlackPearl clients using the .NET or Java SDKs upgrade to these new SDKs so that their clients will be more compatible with BlackPearl 3.0. These new releases should not require any code changes.

BlackPearl 3.0 will be released soon, and shortly thereafter we will be releasing further updates to our SDKs to access the new version 3.0 features. Read about these upcoming additional SDK updates  in our recent blog post Preparing for the BlackPearl 3.0 Release.

Download Latest .NET SDK

Download Latest Java SDK

Download Latest Java CLI

NAB: See BlackPearl, Learn About DS3 Development

Spectra Logic will be at NAB 2015 in Las Vegas, Nevada, USA from April 11-16 in Booth SL 11816 displaying our products and meeting with customers and partners. The Developer Program team will be there in full force to help out developers and spread our message. We’ll be doing demos at our booth and showing how easy it is to get started with DS3 client development. We’ll also show some of the great client applications we have built using our own software development kits (SDKs) that allow users to easily move data to tape. Please stop by and visit us if you are in town.