Spectra Logic

1-800-833-1132 : 1-303-449-6400

Spectra S3 Overview

The BlackPearl Deep Storage Gateway allows data to move seamlessly into deep storage in a way not previously possible. Spectra S3 is the first native REST-based interface to deep storage which enables easy archiving of large amounts of bulk data. It enables users to deploy tape, nearline disk, and online disk storage that is cost effective, easy to manage, and scalable to exabytes of data.

Spectra S3 utilizes the standard Amazon S3 operations plus additional operations specifically designed to optimize the transport of data objects to and from deep storage. The additional operations define the job so that the BlackPearl gateway interacts with the objects efficiently and define the data policy to customize where and for how long specific data is stored.

The first of these additional operations is called START BULK PUT. It is an HTTP PUT operation that provides the BlackPearl gateway with information about the objects that the client wants to send as a single job for storing on tape. The Create Bulk Put command is sent with a payload that is made up of a list of object names and corresponding object sizes. This information allows the BlackPearl gateway to plan the initial storage of the objects in its cache, and how it will store the data on tape. The response to the Create Bulk Put command is a specifically ordered list of how the BlackPearl gateway wants those files (objects) sent. See Processing a Bulk Put Job and Create Bulk Put for details.

The second command is called Create Bulk Get. The Create Bulk Get command is actually an HTTP PUT command because it too contains a payload for the BlackPearl gateway. This payload is a list of objects that the client wants to get from the BlackPearl gateway. It is not necessary for the request payload to contain the size of the files because the BlackPearl gateway already knows the sizes of the objects (files). The response to the request is again an ordered list of the objects and information about the objects, including if they are already in the cache and ready to be retrieved from cache by a GET command.

Knowing the files that the client wants to retrieve, the BlackPearl gateway can make the best use of its resources in retrieving the objects. For example, if the list of objects spans across four different tapes and there are four tape drives available, those four tapes can all be loaded into drives and the objects can be read back in parallel, greatly improving the speed at which the client can get all of the objects. Without the Create Bulk Get request, the client would be asking the BlackPearl gateway for those objects in a less efficient manner. See Processing a Bulk Get Job and Create Bulk Get for details.

Storing large amounts of bulk data on tape has historically presented challenges. Spectra S3 addresses these challenges:

§

Tape drives are sequential block storage devices, with data laid out in a sequential manner along the full length of the tape. This makes it inefficient to retrieve data out of order. Spectra S3 plans and queues a large amount of data to be efficiently written to tape; it logically groups data on tape in a way that reflects how the client is likely to read it back.

§

Because of the mechanical nature of the tape media and drives, tape drives demand a large amount of data to be available, via a fast connection. When data is not efficiently streamed for the tape drives to write (due to slow data buffering or a slow connection to the drive), the result is poor write performance. This poor performance is due to a phenomenon referred to as "shoe-shining". When a drive is sent a small amount of data, it writes the data and then is forced to stop. Because the tape cannot stop instantaneously, the drive overshoots a small amount and the tape is not in position for the next write operation. To compensate, the tape drive rewinds to get back to the correct position for the next write. If the next write also has a small amount of data, then the drive writes the next portion and again stops, overshoots, and rewinds, causing a back and forth "shoe-shining" like action. Spectra S3 caches data on the BlackPearl Deep Storage Gateway before starting the transfer to tape, which prevents the shoe-shining behavior from occurring.

§

Classically, different tape storage devices wrote data to tape in unique ways, locking you into a proprietary and single vendor solution to retrieve previously written data. Spectra S3 writes data to tape using the open source Linear Tape File System® (LTFS). With LTFS, data is always accessible with any LTFS enabled system.