Spectra Logic

1-800-833-1132 : 1-303-449-6400

Create Bulk GET

Description

Create a job to stream GET object requests. See Processing a Bulk GET Job for an overview of the process.

The BlackPearl system will process a maximum of 1,000 concurrent active jobs.

Requests

Syntax

PUT http[s]://{datapathDNSname}/_rest_/bucket/{bucket UUID or name}?operation=START_BULK_GET[&aggregating=TRUE|FALSE]​[&chunk_client_processing_order_guarantee=IN_ORDER|NONE][&implicit_job_id_resolution=TRUE|FALSE]​[&name={string}][&priority=URGENT|HIGH|NORMAL|LOW]

Request Parameters

Parameter	Description	Required
operation	The operation to perform. For this request, the operation is to start a bulk GET. Value: START_BULK_GET	yes
aggregating	Whether or not to aggregate jobs. When selected, if additional GET jobs using the same bucket and chunk_client_processing_order_guarantee are created within 30 minutes of the initial job creation, they are appended to the initial job to create a larger job. Values: TRUE, FALSE (default) Notes:   § Do not use aggregating for jobs larger than 50 GB. § The client must be able to handle the changing job structure to use this feature. § Data is not written to tape storage domains until the 30 minute aggregation time is complete.	no
chunk_client_ processing_order_guarantee	Specifies whether the job chunks must be processed in order. Setting a value of NONE will achieve maximum performance, but requires the client to get chunks as they become available, even if they become available out of order. Values: IN_ORDER, NONE Default: Configured in the data policy for the bucket.	no
implicit_job_id_  resolution	Whether GET requests that are part of this job determine the job ID implicitly (TRUE), or must specify the job ID (FALSE). Implicitly resolving a GET to a job is not always reliable. For example, if two clients GET the same object using two different jobs, implicit resolution may result in the wrong job marking the GET complete. Clients that are aware of the job ID are strongly encouraged to explicitly provide the job ID and leave implicit_job_id_resolution=FALSE. Values: TRUE, FALSE (default)	no
name	A name to assign to the job for tracking.	no
priority	The priority for processing this job. The job priority determines the assigned resources and processing order. Jobs with Priority URGENT can use up all of the resources and prevent other jobs from making progress. Use this priority sparingly. Values: URGENT, HIGH, NORMAL, LOW Default: Configured in the data policy for the bucket.	no

Request Elements

An XML payload, formatted as follows, must be sent to describe the GET job to create:

<Objects
   <Object Name="{string}" Length="{64‑bit integer}"
      Offset="{64‑bit integer}"/>
   ...
</Objects>

where the parameters are defined as follows:

Parameter	Description	Required
Objects	A container for the list of objects.	yes
Object	The container for information about one object.	yes
Name	The name of an object to GET. All objects in the list must be in the same bucket.	yes
Length	The length in bytes to get.	no
Offset	The offset in bytes from the start of the object to start the get.	no

Responses

Response Elements

<MasterObjectList
   Aggregating="TRUE|FALSE"
   BucketName="{string}"
   CachedSizeInBytes="{64‑bit integer}"
   ChunkClientProcessingOrderGuarantee="IN_ORDER|NONE"
   CompletedSizeInBytes="{64‑bit integer}"
   EntirelyInCache="TRUE|FALSE"
   JobId="{string}"
   Naked="TRUE|FALSE"
   Name="{string}"
   OriginalSizeInBytes="{64‑bit integer}"
   Priority="CRITICAL|URGENT|HIGH|NORMAL|LOW|BACKGROUND"
   RequestType="GET"
   StartDate="YYYY-MM-DDThh:mm:ss.xxxZ"
   Status="IN_PROGRESS|COMPLETED|CANCELED"
   UserId="{string}"
   UserName="{string}">
   <Nodes>
      <Node EndPoint="{string}" Id="{string}"/>
   </Nodes>
   <Objects
      ChunkId="{string}"
      ChunkNumber="{32‑bit integer}">
      <Object Id="{string}" InCache="TRUE|FALSE"
         Latest="TRUE|FALSE" Length="{64‑bit integer}"
         Name="{string} "Offset="{64‑bit integer}"
         Version="{64‑bit integer}"/>
      ...
   </Objects>
   ...
</MasterObjectList>

where the response elements are defined as follows:

Parameter	Description
MasterObjectList	The container for the response.
Aggregating	Whether the job can have additional PUTs or GETs appended to it. Jobs aggregate if created with the aggregating request parameter set to TRUE, or if created by the BlackPearl system in response to Amazon S3 PUT and GET requests. Values: TRUE, FALSE
BucketName	The name of the bucket that is acted on by the job request.
CachedSizeInBytes	The amount of data that has been transferred to the cache for this job. For RequestType=GET, this is the amount of data either in cache originally, or loaded into cache from the permanent data store.
ChunkClient ProcessingOrder Guarantee	Specifies whether the job chunks will be processed in order. Values: IN_ORDER, NONE
CompletedSizeIn Bytes	The amount of data that is completely processed for this job. For RequestType=GET, this indicates the amount of data that has been read successfully by the client.
EntirelyInCache	Whether all objects in the job are in the BlackPearl cache. Values: TRUE, FALSE
JobId	The UUID for the job.
Naked	Whether the job was created as the result of a native S3 command (TRUE) or a Create Bulk GET/PUT/VERIFY job command (FALSE). Values: TRUE, FALSE
Name	The name assigned to the job for tracking.
OriginalSizeInBytes	The full size of the job.
Priority	The priority for processing this job. The job priority determines the assigned resources and processing order. Values: CRITICAL, URGENT, HIGH, NORMAL, LOW, BACKGROUND
RequestType	Type of job request. Values: GET
StartDate	The date and time the job was started in the format YYYY-MM-DDThh:mm:ss.xxxZ.
Status	The current status of the job. Values: § IN_PROGRESS — The job is currently running. § COMPLETED — The job completed. § CANCELED — The job was canceled by the user or automatically due to internal timeouts.
UserId	The UUID for the user who initiated the job.
UserName	The username of the user who initiated the job.
Nodes	A container for information about all BlackPearl nodes.
Node	A container for information about a single BlackPearl node.
EndPoint	The IP address or DNS name of the BlackPearl node.
Id	The UUID for the node.
Objects	Container for information about the objects in one chunk.
ChunkId	The UUID for the job chunk.
ChunkNumber	The position of the chunk within the job.
Object	The container for information about a single object.
Id	The UUID for the object.
InCache	Indicates if the object is currently in cache on the BlackPearl Converged Storage System. Values: TRUE, FALSE
Latest	Whether this version of the object is the latest. Values: TRUE, FALSE
Length	The length in bytes of the object.
Name	The name of the object.
Offset	The offset in bytes from the start of the object.
Version	The version number of the object.

Error Response

If one or more objects requested only exist on tapes outside of the library, then the following response is provided:

<Error>
   <Code>CONFLICT</Code>
   <HttpErrorCode>409</HttpErrorCode>
   <Message>
     RPC DataPlanner.createGetJob&lt;60821823&gt; FAILED:
     Some of the data requested is offline (e.g. ejected, offline,
     or quiesced) and must first be brought online. Online:
     [TEST079L7]
   </Message>
   <Resource>/_rest_/bucket/eject</Resource>
   <ResourceId>40727470</ResourceId>
</Error>

Use Get Physical Placement to determine the barcode numbers for the required tapes.

Example

Sample Request

This request creates a GET job for two objects in the bucket “bucket1”.

PUT http://blackpearl-hostname/_rest_/bucket/bucket1​ ?operation=start_bulk_get&name="GetJob" HTTP/1.1

<Objects
   Priority="NORMAL"
   chunkClientProcessingOrderGuarantee="IN_ORDER">
   <Object Name="test.aaf"/>
   <Object Name="T950.tif"/>
</Objects>

Sample Response

HTTP/1.1 200 OK
<MasterObjectList
   Aggregating="FALSE"
   BucketName="bucket1"
   CachedSizeInBytes="0"
   ChunkClientProcessingOrderGuarantee="IN_ORDER"
   CompletedSizeInBytes="0"
   EntirelyInCache="FALSE"
   JobId="78dba377-a02a-4c15-b2a1-412514342f17"
   Naked="FALSE"
   Name="GetJob"
   OriginalSizeInBytes="0"
   Priority="HIGH"
   RequestType="GET"
   StartDate="2015-10-07T22:33:17.000Z"
   Status="IN_PROGRESS"
   UserId="a6a04b3d-960a-4799-9c88-6801f5cf5449"
   UserName="user_name">
   <Nodes>
      <Node EndPoint="blackpearl-hostname"
      Id="aed40ca0-5289-49ca-9cd9-cf5a7559a1db"/>
   </Nodes>
   <Objects
      ChunkId="a909e72e-a96a-4c3e-95a3-f3472e14ae17"
      ChunkNumber="1">
      <Object Id="2ee4f397-c169-4a59-96cc-a07280412f43"
         InCache="FALSE" Latest="TRUE" Length="10"
         Name="o1" Offset="10" Version=1"/>
   </Objects>
</MasterObjectList>