BlackPearl software version 1.2 will be released later this week and should start showing up as an update option in the BlackPearl management web interface next week. The 1.2 update includes enhancements to the BlackPearl management web interface, support for new features in our Deep Storage Browser (formerly DS3 Browser) release version 1.2.1, and a number of bug fixes.

To prepare for this release, we have updated the BlackPearl Simulator to Version 1.2 so you can test out this latest code. The Deep Storage Browser version 1.2.1 is now also available for download.

The Deep Storage Browser is our simple drag-and-drop, FTP-like client for BlackPearl. The new 1.2.1 version of this free, open-source Spectra S3 client has a number of improvements, including:

• Search for objects on BlackPearl, including with wildcards -- percent (%) or underscore (_), like SQL
• Upload/download with arrow/click icons
• Dragging a directory from BlackPearl to local machine no longer results in parent directory also being brought over
• Logging
• Folder delete on BlackPearl
• Multi file delete on BlackPearl

These are the features that were most requested by our users. If you have any other ideas to improve the Deep Storage Browser, please Contact Us or use our Google Group.

We hope you like the new versions of BlackPearl and the Deep Storage Browser.

I have created a new video that shows how to create a demo Windows desktop BlackPearl client application using our .NET/C# Software Development Kit (SDK). Anybody can create this client in less than 15 minutes. You do not need an actual BlackPearl gateway, you can use our BlackPearl simulator. And all the other tools you need are free to download. Give it a try.

Here’s the final Visual Studio project files for the demo client that we build.

BlackPearl acts as a caching gateway in front of Spectra Logic’s tape libraries. Typical client applications will both send groups of files to (PUT) and request groups of files from (GET) BlackPearl. BlackPearl uses “Jobs” to contain and keep track of these individual input/output operations.

BlackPearl is capable of managing many jobs at once. Jobs have a selectable “Priority” for processing the job so that client applications can have some control over the resources assigned to each job. The job priority settings are only relevant when the cache or tape drive resources are constrained, in which case the BlackPearl is said to be “throttled”. If there are no resource constraints, then all jobs will be processed equally.

Files moved by a job are broken into one or more “chunks” for processing. Cache space is required to store each chunk. The job priority determines how many chunks a job can use at any one time. The more chunk cache spaces that are available for a job, the more chunks that can be uploaded or downloaded by the job at any one time. Job priority can also affect how many tape drives a job can utilize at any one time. It takes at least two chunks to efficiently and continuously feed a tape drive.

Job Priority Values & Chunk Allocation

Jobs can be set by client applications to have one of the following priority. For chunk allocation, these priorities are only applicable if the BlackPearl is throttled.

• Low – Low priority jobs get a maximum of four chunk cache spaces at any one time.
• Normal -- (default for PUT) Normal priority jobs get a maximum chunk cache space at any one time of either: (a) eight or (b) two times the number of tape drives, whichever is greater.
• High -- (default for GET) High priority jobs get a maximum chunk cache space at any one time of either: (a) sixteen or (b) three times the number of tape drives, whichever is greater.
• Urgent – Urgent priority jobs get special prioritization. An Urgent job is exempt from any maximum limitations. It can use all available chunk cache space that it requests.

When jobs are requesting cache space for their chunks, and there are only a limited number of spaces available, the job that asks for the spaces first will get them, but no more than the maximums described above.

BlackPearl can also create its own system jobs with Critical and Background priority. These priority values are not available for jobs created by client applications, but you may see them on the Jobs Status Screen on the BlackPearl web interface.

• Critical – The job must be executed immediately and cannot wait. This is typically used for tape drive cleaning operations.
• Background – The job can be done when resources are available. This is typically used for tape inspection and reclamation.

Prioritization for Tape Read/Write Operations

The prioritization for tape drive read and write operations for chunks works in this order of preference:

1. Is the priority of the chunk “Urgent”? If yes, goes before all others (this prioritization preference introduced BlackPearl 3.3)
2. Can the chunk use a tape that’s already in a drive? If yes, it goes before a chunk that can’t.
3. Is the chunk’s priority higher (excluding Urgent priority)? If yes, it goes before lower priority chunks.
4. Can the chunk proceed without allocating another tape to itself? If yes, it goes before chunks requiring another tape.
5. Has the chunk been waiting longer in queue? If yes, it goes before newer chunks.

Note that no job, not even an Urgent priority job, can stop or kill other active tape read or write operations.

Other Ways to Improve Job Performance

If the goal is to improve performance on a PUT operation, then the job’s Write Optimization can be set to “Performance” instead of the default “Capacity”. Using performance means that the job data may be written to more tapes simultaneously, which will allow for faster write performance.

The BlackPearl cache can also be set to its “Performance” configuration rather than its default “Capacity” configuration. This will significantly improve the performance for all jobs. See the “Configure the Cache” section of the BlackPearl User Guide.

The Java Command Line Interface (CLI) is a simple but powerful Spectra S3 client for BlackPearl. As the name implies, it allows users to manage files in BlackPearl using a simple command-line interface. The Java CLI, which is both free and open source, was created by Spectra Logic using our Java Software Development Kit and works on Windows, Mac, and Linux.

We have created a Java CLI Reference and Examples page to assist you with using the Java CLI. We find that one of the most common uses of the Java CLI by our customers is moving files to BlackPearl on some periodic basis using a scheduled task or cron job. Our examples therefore show how to use the Java CLI and 7zip (7zip.org) to zip and move files to BlackPearl. The files can then be retrieved from BlackPearl if needed using the Java CLI or the Deep Storage Browser (formerly DS3 Browser) -- another free and open-source Spectra S3 client created by Spectra Logic.

The Java CLI works like many other command line interfaces. There is a base call (ds3_java_cli) with options to control behavior. There are a total of 13 different commands currently available in the CLI. Here is an example of Java CLI command to move a folder from a server to BlackPearl:

ds3_java_cli -e bp1.acmecorp.com -a amVmZmJy -k iYkqUwcd -c put_bulk -b bucket1 -d “c:\Temp\subdir1”

In this example, the command put_bulk is used to move server directory subdir1 to a bucket called bucket1 on the BlackPearl located at bp1.acmecorp.com. The -a is the BlackPearl user’s S3 Access ID and the -k is the user’s S3 Secret Key.

You can optionally set system environment variables for the S3 Access ID (-a), S3 Secret Key (-k), endpoint (-e) and HTTP proxy (-x, not shown).

There are more examples and information located on the Java CLI Reference and Examples page. As always, if you have any questions or need assistance, please use our Forum or Contact Us.

This week Spectra Logic released a new version of the Spectra Logic Developer Program website.

One reason for launching the new site was to update the look and feel to match our recently updated corporate website. Besides looking great, the new sites work on all devices and have some nice interactive features.

But the main reason for the new version of the Developer site was to make our DS3 client development tools easier to download and access. DS3 is our deep storage protocol, an extension of S3, which allows users to move data to deep storage using simple HTTP commands.  This protocol is made possible by our BlackPearl Deep Storage Gateway.  Now anyone can access all the tools needed to create a DS3 application, including a BlackPearl Simulator, Software Development Kits, sample clients, and documentation.

Additionally, Spectra Logic has decided to make its BlackPearl DS3 software development kits (SDKs) and clients both freely downloadable and open source. This has been done for several reasons:

• To make it as easy as possible for developers, a group which consists of both customers and partners, to develop DS3 clients.
• To allow developers to modify the software for their particular needs, and to contribute those modifications back to the community.
• Developers in the web service and cloud storage communities have an expectation that software tools will be freely available and open source. We want to be a trusted partner in that community.
• Spectra Logic is in the business of selling deep storage hardware. More DS3 clients means that we can provide deep storage hardware to more customers.

We hope that you now find it easier than ever to develop DS3 clients. As always, we look forward to your feedback.

Follow me on Twitter @jeffbr

Spectra Logic has entered a new phase in our company’s history. Our unique products have always required us to be both a hardware and software company. But now, for the first time, we are making it easier to store data on our hardware by providing a software interface available to everyone. This means that customers and partners can build their own applications and clients to move data to Spectra’s deep storage tape libraries. This could greatly reduce the cost and complexity of storing data in long-term archives. But it also means that Spectra must provide the tools and resources necessary for users to build these software applications.

That’s where I come in. I am Jeff Braunstein, the new “Developer Evangelist” at Spectra Logic. After working at Spectra for more than 13 years, I am excited to begin my new role and to help our customers and partners. As Spectra’s Developer Evangelist, my job is to be the “Guinea Pig” developer for the Spectra Logic Developer Program. I will be responsible for building clients and tools using our developer resources, ensuring that necessary documentation and examples are provided to build applications, and answering developer questions or providing someone who can. I will work with our Engineering team to make sure our software interface has all the features the developer community needs, and will reach out to the current and potential developer community to demonstrate how to get started and build clients.

Our new software interface is made possible by our BlackPearl™ Deep Storage Gateway. The BlackPearl appliance sits in front of our deep storage tape libraries and allows users to move data using simple HTTP commands. Our command set is called DS3 and is an extension of the S3 storage interface popularized by Amazon®. We provide not only the Application Program Interface (API) for these DS3 commands, but also Software Development Kits (SDKs) in several programming languages to make it even easier to create DS3 clients and applications.

So what are your long-term storage challenges? Could this new, easy way to move data to deep storage help solve your organization’s data storage problems?

Send your feedback and questions my way at jeffbr@spectralogic.com or put them in the comments below. I look forward to hearing from you.

Visit the following links to learn more about Spectra Logic’s Developer Program, our BlackPearl™ Deep Storage Gateway, and our DS3 Software Interface.

Follow me on Twitter @jeffbr