Our Engineering Team has released new versions of the .NET/C# and Java Software Development Kits (SDK), as well as the Java Command-Line Interface (CLI). There are a number of new features and bug fixes in these releases. Probably the most important new feature in these releases is improved support for the upcoming BlackPearl 3.0 release. We strongly recommend that developers currently building BlackPearl clients using the .NET or Java SDKs upgrade to these new SDKs so that their clients will be more compatible with BlackPearl 3.0. These new releases should not require any code changes.
BlackPearl 3.0 will be released soon, and shortly thereafter we will be releasing further updates to our SDKs to access the new version 3.0 features. Read about these upcoming additional SDK updates in our recent blog post Preparing for the BlackPearl 3.0 Release.
BlackPearl uses Bulk PUT and GET commands to move files to and from its storage targets -- tape and disk. We call them “Bulk” commands because they transfer multiple files in one “job”. The bulk commands provide several advantages over traditional, single-file S3 PUT and GET operations, including: 1) ensuring that the BlackPearl cache is ready to receive the files; and 2) providing adequate data to continuously write or read data on to the storage target at a high level of performance.
Many factors affect the file transfer performance to and from BlackPearl, including network configuration and equipment specifications, transfer rate capabilities from the primary storage medium, file transfer software architecture and file sizes/job size. To maximize file transfer performance (typically measured in megabytes per second) on a bulk PUT or GET job, the size of both the individual files in the job and the total job must be considered. Spectra Logic has performed considerable testing and analysis of various files sizes and has determined ideal file sizes for maximizing performance in bulk PUT and GET jobs. The results of this testing are below.
We recommend developers build BlackPearl client that target at least the “Good” numbers below. Based on this recommendation, developers will want individual files sizes in the tens of megabytes and total job sizes in the hundreds of gigabytes. Sizes smaller than these can result in significant performance degradation.
For more tips on building a BlackPearl client, see our Guidance and Tips page.
Performance descriptions:
Poor — Performance will have degraded by no more than one order of magnitude
Fair — Performance will have degraded by around half
Good — Performance will have degraded by around 10%
Great — Maximum/optimal performance
As mentioned in our last two blog posts (see Part 1 and Part 2), BlackPearl 3.0 will be released soon and will include some exciting new features. Shortly after BlackPearl 3.0 is released, we will be also be releasing associated updates to our Software Development Kits (SDKs), which are available in Java, C#/.NET, C and Python. BlackPearl client developers should understand how to prepare for this new release.
For the most part, BlackPearl clients using the existing 1.x SDKs should function normally with BlackPearl 3.0. The one exception to this is the 1.x Java SDK, which will require a small patch in order to work with BlackPearl 3.0. We will be releasing this patch, which will be Java SDK Release 1.2.1, soon and will announce it on this Blog once it becomes available. Also, if your client programmatically creates buckets, you should ensure that the user account the client uses to create the bucket has a default data policy (see screen image below).
When BlackPearl 3.0 is released, we will also be releasing a new version of the BlackPearl Simulator so that developers can test their client code against the new release before upgrading any actual BlackPearl systems. We are recommending this because there have been significant changes and additions to the 3.0 Application Program Interface (API) and, while it is our goal to be fully backward compatible with 1.x clients, client integration testing to ensure your client’s full compatibility with BlackPearl 3.0 would be wise.
The SDKs provide a layer of abstraction over the HTTP-based, RESTful API commands of the BlackPearl. The current 1.x SDKs provide access to only a subset of the most popular BlackPearl API commands. This is because each API command had to be manually programmed in the SDK by our Engineering Team, and it was too time intensive to write SDK commands for all API commands. So while the 1.x SDKs have commands for common actions such as moving files to and from BlackPearl and creating buckets, they did not have commands for less common actions such as inspecting and ejecting tapes.
With our 3.0 SDKs, we developed a technique to automatically generate most of the SDK code needed for each associated API command. This means that almost every BlackPearl API command will have an associated command in each of the SDKs. So developers using the SDKs will now be able to access nearly all API commands via the SDKs, including new features such as Advanced Bucket Management and Access Control Lists. We will also continue to make the SDKs easier to use as we have already done, such as adding the “helper” functions (currently available in the Java and .NET SDKs) that further simplify the archive and restore process.
There are three areas of change that developers should be aware of when upgrading their client from 1.x to 3.0:
Due to the nature of the 3.0 SDKs, and the fact that they support nearly all BlackPearl API commands, we had to restructure the SDK code base compared to the 1.x SDKs. The BlackPearl API supports Standard S3 as well as Spectra S3 commands, and in some cases there are separate commands for each protocol (Standard S3 versus Spectra S3) that do essentially the same thing. So for example, there is a Standard S3 command to create a bucket and a separate Spectra S3 command to create a bucket, each with different input parameters. In order to make both of these commands available in the SDKs, we had to create separate namespaces for each protocol. Therefore, if you are updating your code from 1.x to 3.0, you will likely have to append the appropriate namespace in your code to each command. We will provide specific examples once the 3.0 SDKs are released.
Some method names will have to be changed between 1.x and 3.0. We will provide a full list of those name changes.
There will be a few methods where the ordering of arguments will have to be changed. We will provide a list of of those methods.
The new 3.0 SDKs will also include a new set of documentation and code examples.
Developers should make sure that they are planning for the BlackPearl 3.0 release. Those upgrading from 1.x to 3.0 and who are not changing client functionality should expect it to be straightforward. For those wanting to enhance their clients to take advantage of new 3.0 features, our Developer website will be available for help. Look for more information on this Blog once BlackPearl 3.0 is released.
