What is DS3 (formerly Spectra S3)?
The DS3 interface of BlackPearl incorporates Deep Storage REST extensions with the Amazon Web Services S3 API to enable bulk read or write operations of data objects to and from a storage medium. More on DS3>
What is BlackPearl?
The Spectra Logic BlackPearl Appliance provides Deep Storage Gateway Services with tape, archive disk, and public cloud. The BlackPearl Gateway facilitates where and how DS3 (formerly Spectra S3) buckets are organized, categorized, and placed on the storage target. BlackPearl individual tapes will be formatted with the Linear Tape File System (LTFS), and are readable by any LTFS-compatible system. More on BlackPearl>
How are users created on BlackPearl?
Users are created using the BlackPearl web interface. Users can create one or more “buckets” associated with their user account in which to store their objects/files. Users are assigned an ID and Key which they use for authentication when issuing DS3 (formerly Spectra S3) commands.
What happens when I delete a file? Is the file deleted on tape?
When you issue the DS3 (formerly Spectra S3) command to delete a file, it is deleted from the BlackPearl cache if it exists. However, it is not deleted from tape. In order to delete a file from tape, you must delete all files in the bucket and the bucket itself (assuming bucket-isolated tapes). All tapes associated with that bucket will then be reformatted and be made available to the general pool. BlackPearl will also, in the background, reformat any tapes that contain only deleted files. When a tape is reformatted it is not fully wiped/erased from start to finish. Only the file index data is deleted.
How do I know when a tape library or tape partition is full?
There are two ways to learn about capacity. The first way: The BlackPearl web interface includes a tape management dashboard. This dashboard lists all tapes in the library or partition and includes capacity and usage information on each. The second way: the DS3 (formerly Spectra S3) Command Set includes a “Get Tapes” command. This command will retrieve a list of all tapes and the capacity and available space on each. A client application could use this data to identify when a tape library or partition is near capacity.
How does BlackPearl process multiple jobs? How does it prioritize jobs?
Please see our blog post on BlackPearl Spectra S3 Job Priority
What is the maximum file size that can be moved using DS3 (formerly Spectra S3)?
There is no limitation to the file size that can be used with DS3 using the bulk DS3 commands. Files larger than a certain size (default 64GB) will be broken into multiple chunks.
For the Bulk PUT and GET commands, will the “chunks” go to/from BlackPearl in the specified order?
When you do a Bulk PUT to upload objects to BlackPearl, BlackPearl will provide a list of chunks ready to be uploaded. You must send the chunks to BlackPearl in order. When you do a Bulk GET to retrieve objects, you can specify whether you want BlackPearl to retrieve the objects in the order BlackPearl specifies. Putting the objects in order may be more convenient for the client but may also cause BlackPearl to take longer to retrieve the objects.
Where is metadata stored for the objects I upload to BlackPearl?
BlackPearl and DS3 (formerly Spectra S3) allow users to add metadata to the objects that are uploaded. The metadata is name-value pairs and can be used for such things as file permissions, content information, tagging, and more. The metadata for each object is stored in two locations:
·In the object database, which resides on BlackPearl.
·When the object has moved to tape, in the LTFS extended attributes for that object.
Does BlackPearl support checksums?
Yes, you can use checksums to ensure that your data was successfully transmitted to BlackPearl. You can use MD-5, CRC-32, CRC-32C, SHA-256, or SHA-512 checksums.
Do I need to worry about tape library compression or encryption for my DS3 (formerly Spectra S3) application?
Spectra Logic tape libraries offer data compression and encryption to optimize and secure data. Tape library compression occurs automatically, and users can also choose to activate tape library encryption. Fortunately, developers building a DS3 (formerly Spectra S3) client do not need to worry about these compression or encryption features. The tape-library-level compression and encryption features are transparent to the DS3 service. All DS3 communication and data will be provided uncompressed and unencrypted. The developer can choose to use https/SSL (port 443) to encrypt communication at the HTTP level. And, of course, applications can pass pre-compressed and/or pre-encrypted data to the DS3 service. But the tape-library-level compression and encryption will be transparent to the DS3 client.
Does BlackPearl support WORM tape media?
Currently, BlackPearl/DS3 (formerly Spectra S3) does not support Write Once Read Many (WORM) tape media.
I have completed initial testing of my DS3 (formerly Spectra S3) client with the BlackPearl simulator. How do I test it on a real BlackPearl?
Once you have completed initial testing, we can arrange for you to have access to a real BlackPearl to further test your client. Please Contact Us to arrange this access.
Can I sell the BlackPearl client that I have developed with Spectra SDKs?
The Apache license we use for the SDKs allows them to be used in a commercial product without it being open source. See the Apache License Version 2.0.
How do I monitor the HTTP traffic generated by my application?
If you are using the SDK’s for application development, you may want to monitor the HTTP traffic generated by your application. Several of us at Spectra use zaproxy for this purpose. It can be used as a proxy server for your application so that you can see all HTTP traffic created by your application in zaproxy.
Does BlackPearl support the Standard S3 protocol?
DS3 (formerly Spectra S3) is an extension of the Standard S3 HTTP storage protocol. BlackPearl does not support the Standard S3 protocol.
Do the DS3 (formerly Spectra S3) SDKs use existing S3 code libraries?
No, the DS3 Software Development Kits were written by Spectra Logic. The DS3 SDKs do not include any other S3 code libraries.
Can BlackPearl read tapes written by other systems/software?
Yes, BlackPearl can read and import most tapes written in the standard LTFS format. It cannot read non-LTFS tapes.
What is the maximum number of objects that can be moved in a Bulk GET or PUT operation?
The DS3 (formerly Spectra S3) Bulk PUT and Bulk GET commands support up to 500,000 objects in one operation. Learn more about DS3 bulk operations.
How do I move files from the BlackPearl simulator to the parent/primary operating system?
The BlackPearl simulator runs as BSD Unix in a Virtual Machine (VMDK). We use VirtualBox VM for running the simulator. What if you need to move files from the simulator to the parent/primary operating system? If you are using a *nix system, you can use the SCP command. If you are on Windows, you can use the program WinSCP, which is a program that allows you to copy files from a *nix machine to your Windows computer in an FTP-like interface. This could be a great tool particularly if you are asked by the Spectra Engineering Team to provide log files from the simulator.
Spectra RioBroker Questions
Why was RioBroker created? What is its purpose?
Spectra Logic has provided SDKs that allow partners to integrate directly with BlackPearl. The benefit of RioBroker is that it provides a layer of abstraction from BlackPearl and greatly simplifies the interface and integration. Essentially, rather than use SDKs to access BlackPearl directly, we are now providing SDKs to integrate with RioBroker. The result is a stable API which will not change even if BlackPearl has new releases.
Will my current BlackPearl integration continue to work?
If you have an existing BlackPearl integration, it will continue to work. You do not have to migrate to RioBroker immediately. That said, RioBroker is the new interface to BlackPearl and will be the focus of future development by Spectra Logic.
Will there be ongoing releases of the RioBroker?
Spectra Logic will put out new releases of RioBroker as needed. That said, the API to RioBroker is expected to be extremely stable and unchanging which means partners will not have to “re-write” the integration with their software.
Will new releases of BlackPearl require me to update my RioBroker integration?
No, because RioBroker sits between the partner software and the BlackPearl, we are able to “hide” any changes made to the BlackPearl interface. That said, to be compatible with a new BlackPearl release, it may be necessary to upgrade to the latest RioBroker release.
What is the benefit of providing a single namespace?
A single namespace simplifies management to one interface. When working with multiple file systems, a single namespace represents all the data as if it were in the same location.