The Spectra Simple Storage Service (formerly DS3) interface incorporates Deep Storage REST extensions with the Amazon Web Services S3 API to enable bulk read or write operations of data objects to and from storage medium. More on Spectra S3>
The Spectra Logic BlackPearl Appliance provides Deep Storage Gateway Services with tape, archive disk, and public cloud. The BlackPearl Gateway facilitates where and how Spectra S3 buckets are organized, categorized, and placed on the storage target. BlackPearl individual tapes will be formatted with the Linear Tape File System (LTFS), and are be readable by any LTFS-compatible system. More on BlackPearl>
Users are created using the BlackPearl web interface. Users can create one or more “buckets” associated with their user account in which to store their objects/files. Users are assigned an ID and Key which they use for authentication when issuing Spectra S3 commands.
When you issue the Spectra S3 command to delete a file, it is deleted from the BlackPearl cache if it exists. However, it is not deleted from tape. In order to delete a file from tape, you must delete all files in the bucket and the bucket itself (assuming bucket-isolated tapes). All tapes associated with that bucket will then be reformatted and be made available to the general pool. BlackPearl will also, in the background, reformat any tapes that contain only deleted files. When a tape is reformatted it is not fully wiped/erased from start to finish. Only the file index data is deleted.
There are two ways to learn about capacity. The first way: The BlackPearl web interface includes a tape management dashboard. This dashboard lists all tapes in the library or partition and includes capacity and usage information on each. The second way: the Spectra S3 Command Set includes a “Get Tapes” command. This command will retrieve a list of all tapes and the capacity and available space on each. A client application could use this data to identity when a tape library or partition is near capacity.
Please see our blog post on BlackPearl Spectra S3 Job Priority
There is no limitation to the file size that can be used with Spectra S3 using the bulk Spectra S3 commands. Files larger than a certain size (default 64GB) will be broken into multiple chunks.
When you do a Bulk PUT to upload objects to BlackPearl, BlackPearl will provide a list of chunks ready to be uploaded. You must send the chunks to BlackPearl in order. When you do a Bulk GET to retrieve objects, you can specify whether you want BlackPearl to retrieve the objects in the order BlackPearl specifies. Putting the objects in order may be more convenient for the client but may also cause BlackPearl to take longer to retrieve the objects.
BlackPearl and Spectra S3 allow for users to add metadata to the objects that are uploaded. The metadata are name-value pairs, and can be used for such things as file permissions, content information, tagging, and more. The metadata for each object is stored in two locations:
·In the object database, which resides on BlackPearl.
·When the object has moved to tape, in the LTFS extended attributes for that object.
Yes, you can use checksums to ensure that your data was successfully transmitted to BlackPearl. You can use MD-5, CRC-32, CRC-32C, SHA-256, or SHA-512 checksums.
Spectra Logic tape libraries offer data compression and encryption to optimize and secure data. Tape library compression occurs automatically, and users can also choose to activate tape library encryption. Fortunately, developers building a Spectra S3 client do not need to worry about these compression or encryption features. The tape-library-level compression and encryption features are transparent to the Spectra S3 service. All Spectra S3 communication and data will be provided uncompressed and unencrypted. The developer can choose to use https/SSL (port 443) to encrypt communication at the http level. And, of course, applications can pass pre-compressed and/or pre-encrypted data to the Spectra S3 service. But the tape-library-level compression and encryption will be transparent to the Spectra S3 client.
Currently BlackPearl/Spectra S3 does not support Write Once Read Many (WORM) tape media.
Once you have completed initial testing, we can arrange for you to have access to a real BlackPearl to further test your client. Please Contact Us to arrange this access.
The Apache license we use for the SDKs allows them to be used in a commercial product without it being open source. See the Apache License Version 2.0.
If you are using the SDK’s for application development, you may want to monitor the HTTP traffic generated by you application. Several of us at Spectra use zaproxy for this purpose. It can be used as a proxy server for your application so that you can see all HTTP traffic created by your application in zaproxy.
No, the Spectra S3 Software Development Kits were written by Spectra Logic. The Spectra S3 SDKs do not include any other S3 code libraries.
Yes, BlackPearl can read and import most tapes written in the standard LTFS format. It cannot read non-LTFS tapes.
The Spectra S3 Bulk PUT and Bulk GET commands support up to 500,000 objects in one operation. Learn more about Spectra S3 bulk operations.
The BlackPearl simulator runs as BSD Unix in a Virtual Machine (VMDK). We use VirtualBox VM for running the simulator. What if you need to move files from the simulator to the parent/primary operating system? If you are using a *nix system, you can use the SCP command. If you are on Windows, you can use the program WinSCP, which is a program that allows you to copy files from a *nix machine to your Windows computer in a FTP-like interface. This could be a great tool particularly if you are asked by the Spectra Engineering Team to provide log files from the simulator.