The Spectra Simple Storage Service (formerly DS3) interface incorporates Deep Storage REST extensions with the Amazon Web Services S3 API to enable bulk read or write operations of data objects to and from storage medium. The first implementation of Spectra S3 leverages tape for bulk object storage operations to achieve low storage costs with infinite scalability. More on Spectra S3>
The Spectra Logic BlackPearl Appliance provides Deep Storage Gateway Services with tape, archive disk, and public cloud. The BlackPearl Gateway facilitates where and how Spectra S3 buckets are organized, categorized, and placed on the storage target. BlackPearl individual tapes will be formatted with the Linear Tape File System (LTFS), and will be readable by any LTFS-compatible system. More on BlackPearl>
Users are created using the BlackPearl web interface. Users can create one or more “buckets” associated with their user account in which to store their objects/files. Users are assigned an ID and Key which they use for authentication when issuing Spectra S3 commands.
When you issue the Spectra S3 command to delete a file, it is deleted from the BlackPearl cache if it exists. However, it is not deleted from tape. In order to delete a file from tape, you must delete all files in the bucket and the bucket itself (assuming bucket-isolated tapes). All tapes associated with that bucket will then be reformatted and be made available to the general pool. BlackPearl will also, in the background, reformat any tapes that contain only deleted files. When a tape is reformatted it is not fully wiped/erased from start to finish. Only the file index data is deleted.
There are two ways to learn about capacity. The first way: The BlackPearl web interface includes a tape management dashboard. This dashboard lists all tapes in the library or partition and includes capacity and usage information on each. The second way: the Spectra S3 Command Set includes a “Get Tapes” command. This command will retrieve a list of all tapes and the capacity and available space on each. A client application could use this data to identity when a tape library or partition is near capacity.
If a client is using the bulk Spectra S3 PUT commands, BlackPearl will wait until all “chunks” that are part of the bulk PUT job have been moved from the client to BlackPearl cache. Once it has received all chunks that are part of the job, it will start moving them to tape. For a traditional/standard S3 PUT, where just one object is sent to BlackPearl, BlackPearl will wait for up to 30 minutes to see if it receives any other objects from clients. If it does not after 30 minutes, it will go ahead and write the object to tape. In all cases, if the object or chunk exceeds 100GB in size, it will start moving that object/chunk to tape immediately. Note that if a new tape needs to be formatted, it could take an additional 6 minutes before BlackPearl starts writing data to tape.
When you do a Bulk PUT to upload objects to BlackPearl, BlackPearl will provide a list of chunks ready to be uploaded. You must send the chunks to BlackPearl in order. When you do a Bulk GET to retrieve objects, you can specify whether you want BlackPearl to retrieve the objects in the order BlackPearl specifies. Putting the objects in order may be more convenient for the client but may also cause BlackPearl to take longer to retrieve the objects.
BlackPearl and Spectra S3 allow for users to add metadata to the objects that are uploaded. The metadata are name-value pairs, and can be used for such things as file permissions, content information, tagging, and more. The metadata for each object is stored in two locations:
·In the object database, which resides on BlackPearl.
·When the object has moved to tape, in the LTFS extended attributes for that object.
Spectra Logic tape libraries offer data compression and encryption to optimize and secure data. Tape library compression occurs automatically, and users can also choose to activate tape library encryption. Fortunately, developers building a Spectra S3 client do not need to worry about these compression or encryption features. The tape-library-level compression and encryption features are transparent to the Spectra S3 service. All Spectra S3 communication and data will be provided uncompressed and unencrypted. The developer can choose to use https/SSL (port 443) to encrypt communication at the http level. And, of course, applications can pass pre-compressed and/or pre-encrypted data to the Spectra S3 service. But the tape-library-level compression and encryption will be transparent to the Spectra S3 client.
If you are using the SDK’s for application development, you may want to monitor the HTTP traffic generated by you application. Several of us at Spectra use zaproxy for this purpose. It can be used as a proxy server for your application so that you can see all HTTP traffic created by your application in zaproxy.
Spectra S3 (formerly DS3) is an extension of the Standard S3 HTTP storage protocol. If a client will be archiving to tape via BlackPearl, it is imperative that the Spectra S3 protocol is used. If a client is archiving to disk via BlackPearl (using ArcticBlue), the Standard S3 protocol may be used, as long as a copy of all data is always available on disk. To learn more, read BlackPearl Support for Standard S3 Compatible Clients.
The BlackPearl simulator runs as BSD Unix in a Virtual Machine (VMDK). We use VirtualBox VM for running the simulator. What if you need to move files from the simulator to the parent/primary operating system? If you are using a *nix system, you can use the SCP command. If you are on Windows, you can use the program WinSCP, which is a program that allows you to copy files from a *nix machine to your Windows computer in a FTP-like interface. This could be a great tool particularly if you are asked by the Spectra Engineering Team to provide log files from the simulator.