On October 15, 2015, Spectra Logic announced ArcticBlue, new nearline disk solution that sits behind the BlackPearl Deep Storage Gateway. BlackPearl now provides an S3 object private cloud interface to the following storage products:
- Spectra Logic tape libraries – BlackPearl has supported archive to tape libraries since its original release
- ArcticBlue – ArcticBlue is a new nearline storage target for BlackPearl. Read more about ArcticBlue
- Spectra Online Disk – Spectra Online Disk with Enterprise SAS drives are also a new storage target for BlackPearl
As part of the ArcticBlue release in December, we will also be releasing the next major software version of BlackPearl, Version 3.0 (we are skipping 2.0 to get BlackPearl and Verde on a common code base release). This new version will not only include support for ArcticBlue and Spectra Online Disk, but also includes two other new major features:
- Advanced Bucket Management – Allows data policies to be set on buckets to control how many copies and for how long objects are stored on each storage product listed above. Advanced Bucket Management is covered below.
- Access Control Lists – Provides sophisticated permission control on objects and buckets. Access Control Lists will be covered in Part 2 of this blog post.
Advanced Bucket Management
Advanced Bucket Management (ABM) is an extremely powerful new feature provided at no additional cost in BlackPearl Version 3.0. Policies are set on buckets that determine which storage type each object in the bucket will be stored on and for how long each object will be stored on each storage type. You can see an example scenario in the diagram below. Though this is probably not a realistic scenario, it does show all the different policy options.
In the diagram above, Bucket 3 has a 4-copy data policy. When objects are moved to Bucket 3, a copy of the object is immediately placed in each of the four storage domains:
- A copy will be placed on online disk for 30 days for very fast object retrieval.
- A copy will be placed on ArcticBlue nearline disk for 2 years for fairly fast object retrieval (ArcticBlue is “power down” disk so it takes a bit longer to respond than online disk).
- A copy will be placed in a Spectra T950 tape library with TS1150 tape drives. This copy has no expiration.
- A copy will be placed in a Spectra T200 tape library with LTO-7 tape drives. The tapes on which this object is stored will be ejected from the library for offsite storage.
When an object stored in Bucket 3 is requested by an application, the BlackPearl knows to retrieve it from the fastest available storage domain. So if the object is being requested within the first 30 days, it will be retrieved from online staging disk. Between 31 days and 2 years, the object will be retrieved from ArcticBlue nearline disk. And after two years the object will be retrieved from tape.
When a bucket is created, it must now be assigned a data policy. In the web management interface you will be forced to choose a data policy (see below). If you create a bucket via the API/SDK, you can also assign a data policy. But if you don’t assign a data policy, the user’s default data policy will be assigned to the bucket.
NOTE: If you are upgrading from 1.x to 3.0, you will need to assign a default data policy for each user.
BlackPearl will ship with a number of common data policies, as shown on the screen image above. These policies are automatically created based on the hardware attached. If only tape is attached then two tape policies will be auto generated and will work for most users. However, users can create their own data policies as well. Developers will be able to manipulate nearly all aspects of data policy management via the API and SDKs. We will be providing documentation on how to do this as we get closer to the release date of BlackPearl 3.0 in December.
To support Advanced Bucket Management at the most basic level in a BlackPearl client, the client should support the ability to use multiple buckets. Having multiple buckets, as shown above, will allow for the user to choose different data policies. One policy for frequently accessed data could have one copy in Spectra Online Disk for 120 days and one copy in ArcticBlue Nearline Disk forever. A second policy could be the “One Copy Tape, One Copy Nearline Disk” which is two permanent copies, great for warm data that needs parallel access while providing genetic diversity with extremely high level of durability. This would provide users two different types of storage profiles within one platform.
You can learn more about the new features of BlackPearl by viewing the recording of our inaugural Developer Summit.
In Part 2 of this blog post, I will describe the new Access Control Lists (ACLs) feature.