BlackPearl uses Bulk PUT and GET commands to move files to and from its storage targets -- tape and disk. We call them “Bulk” commands because they transfer multiple files in one “job”. The bulk commands provide several advantages over traditional, single-file S3 PUT and GET operations, including: 1) ensuring that the BlackPearl cache is ready to receive the files; and 2) providing adequate data to continuously write or read data on to the storage target at a high level of performance.
Many factors affect the file transfer performance to and from BlackPearl, including network configuration and equipment specifications, transfer rate capabilities from the primary storage medium, file transfer software architecture and file sizes/job size. To maximize file transfer performance (typically measured in megabytes per second) on a bulk PUT or GET job, the size of both the individual files in the job and the total job must be considered. Spectra Logic has performed considerable testing and analysis of various files sizes and has determined ideal file sizes for maximizing performance in bulk PUT and GET jobs. The results of this testing are below.
We recommend developers build BlackPearl client that target at least the “Good” numbers below. Based on this recommendation, developers will want individual files sizes in the tens of megabytes and total job sizes in the hundreds of gigabytes. Sizes smaller than these can result in significant performance degradation.
For more tips on building a BlackPearl client, see our Guidance and Tips page.
Individual Objects/Files in a Bulk PUT or GET Job
[efstable width =”100%”]
[efstable_head]
[efsth_column]Performance[/efsth_column]
[efsth_column]File Size[/efsth_column]
[/efstable_head]
[efstable_body]
[efstable_row]
[efsrow_column]Poor[/efsrow_column]
[efsrow_column]>= 2 MB[/efsrow_column]
[/efstable_row]
[efstable_row]
[efsrow_column]Fair[/efsrow_column]
[efsrow_column]>= 5 MB[/efsrow_column]
[/efstable_row]
[efstable_row]
[efsrow_column]Good[/efsrow_column]
[efsrow_column]> = 50 MB[/efsrow_column]
[/efstable_row]
[efstable_row]
[efsrow_column]Great[/efsrow_column]
[efsrow_column]>= 1 GB[/efsrow_column]
[/efstable_row]
[/efstable_body]
[/efstable]
Total Bulk Job Size
[efstable width =”100%”]
[efstable_head]
[efsth_column]Performance[/efsth_column]
[efsth_column]LTO-6 Tape Drives[/efsth_column]
[efsth_column]LTO-7 or TS1150 Tape Drives[/efsth_column]
[/efstable_head]
[efstable_body]
[efstable_row]
[efsrow_column]Poor[/efsrow_column]
[efsrow_column]>= 10 GB[/efsrow_column]
[efsrow_column]>= 20 GB[/efsrow_column]
[/efstable_row]
[efstable_row]
[efsrow_column]Fair[/efsrow_column]
[efsrow_column]>= 50 GB[/efsrow_column]
[efsrow_column]>= 100 GB[/efsrow_column]
[/efstable_row]
[efstable_row]
[efsrow_column]Good[/efsrow_column]
[efsrow_column]>= 200 GB[/efsrow_column]
[efsrow_column]>= 400 GB[/efsrow_column]
[/efstable_row]
[efstable_row]
[efsrow_column]Great[/efsrow_column]
[efsrow_column]>= 1 TB[/efsrow_column]
[efsrow_column]>= 2 TB[/efsrow_column]
[/efstable_row]
[/efstable_body]
[/efstable]
Performance descriptions:
Poor — Performance will have degraded by no more than one order of magnitude
Fair — Performance will have degraded by around half
Good — Performance will have degraded by around 10%
Great — Maximum/optimal performance