Software applications must be able to handle unexpected conditions. BlackPearl client developers must consider what unexpected conditions and exceptions may occur in their clients and have the client respond to them appropriately. This article provides information on some of the common unexpected conditions in BlackPearl clients and how to address them.
Using SDKs and API
If the BlackPearl client uses one of the BlackPearl SDKs, unexpected conditions may result in the SDK methods throwing an exception. Most of the SDK Documentation includes information on the exceptions that are thrown by each method. Developers should ensure that their BlackPearl clients can properly handle the exceptions thrown by each method being used.
If the client communicates with the BlackPearl API directly, and does not use SDKs, then developers can use the API Documentation to review the possible HTTP response codes for each operation. Developers should ensure that their BlackPearl clients properly handle all response codes.
BlackPearl Authentication Exceptions
A client connecting to BlackPearl must be authenticated. This authentication information must be provided by the administrator or user of the client. When connecting to a BlackPearl, a user typically provides a client with the BlackPearl server name or IP address, the user Access ID, and user Secret Key. Some clients may also require users to provide a BlackPearl bucket name.
Figure 1. Login screen of the Deep Storage Browser with invalid Access ID or Secret Key.
Clients should ensure that users provide valid connection information. Clients should not save the connection information until the client has contacted the BlackPearl and ensured that all connection information is valid. Developers may also choose to provide a “Test” button to allow users to test the connection information before attempting to save it. Developers should test their clients to ensure that if invalid connection information is provided, the information is not saved and the user receives a useful message to correct the connection problem.
The Java SDK “Get Service” sample code, which shows how to get a list of buckets, also includes examples for handling invalid Access IDs, Secret Keys, and server names or IP addresses. View Get Service Example
Object Already Exists/Doesn’t Exist
When sending a file to BlackPearl, if a file with the same name already exists in the bucket, by default BlackPearl will not allow this. BlackPearl will reject the newer file, and the client will receive an exception or error. Clients should be prepared to handle this scenario of a file upload failing due to the file already existing. Developers should test this scenario in their application.
In some cases, a BlackPearl client may try to retrieve a file from BlackPearl that the client believes exists on BlackPearl but in fact does not. This missing file could be caused by various scenarios, including a different client accessing the BlackPearl and deleting the file. Clients should be able to handle the scenario of a file unexpectedly not existing in BlackPearl and be able to provide a useful message to the user. Developers should test their client for the missing file use case by trying to restore a file from BlackPearl that doesn’t exist. A different BlackPearl client, such as the Eon Browser, can be used to delete the file for this test scenario.
BlackPearl Cache Full
BlackPearl contains a disk cache that is used to temporarily store files. Files sent to BlackPearl are first placed in this cache before being moved to their permanent storage location. In some situations, such as a tape library or other storage target being unexpectedly offline, it is possible that this cache could fill up and not be able to receive more files.
BlackPearl clients must be able to handle the condition of the cache being full. The client should attempt some limited amount of retries, with pauses between each retry, to continue to try to write the files to BlackPearl’s cache. If the files still cannot be written after these retries, the client should provide a useful message to the user to alert them that the cache is full and direct them to inspect the BlackPearl system. We recommend that the message should be presented to the user after about 60 minutes of trying to write to the cache. You may want to use a different time depending on your application.
The .NET/C# and Java SDKs include Helper classes that simplifies the process of moving files to and from BlackPearl. If a client is using the Helper classes to transfer files, the optional retryAfter parameter should be set with a maximum number after which the user should be notified. If the client isn’t using the Helper classes, it will be using the Get Job Chunks Ready for Processing command to determine which files in a job are ready to be transferred to BlackPearl’s cache. If the command repeatedly indicates that no chunks are ready to be transferred, it is an indication that the BlackPearl cache may be full. In this case clients should alert the user to check the BlackPearl system.
Developer should test their client to ensure it gracefully handles the full cache scenario. Our Certification Program Test Plan provides details on how to fill the BlackPearl cache to test this scenario.
Connection Problems and Job Recovery
A BlackPearl client application may experience problems when transferring files if there are network connectivity issues or if the BlackPearl is down or unresponsive. A BlackPearl client must be able to properly respond and recover from these problems.
The job.transfer() method in the Helper classes that transfers the files between the client application and BlackPearl will throw an exception if there are any interruptions during the transfer of the files in a job. The class includes recoverWriteJob() and recoverReadJob() methods that should be used to attempt to restart and complete the transfer job. These methods will automatically determine which files remain to be transferred in the job and attempt to complete the transfer of the remaining files and/or file parts. These recovery methods should only be tried a handful of times (with some wait between each try) before alerting the user to the problem so that they can attempt to fix it (i.e. check status of network and BlackPearl). We have code samples in the Java SDK that shows how to recover jobs using these recover methods. See Recover Read Job Code Example 1, Recover Read Job Code Example 2, Recover Write Code Example 1.
Developers should test their client to make sure it can recover from interrupted jobs. The easiest method to test this interruption is to disconnect the Ethernet cable on BlackPearl data port during an active job.
Some other notes about job recovery:
- If the client isn’t using the Helper classes, it can call the getJob() method to determine which files in a job have and have not been transferred. The client can then attempt to transfer the remaining files.
- A client may choose to poll the BlackPearl using the Verify System Health method to verify that the BlackPearl is online and to get basic health information about the BlackPearl. See Verify System Health in Java SDK.
- If a PUT/write job to BlackPearl is not finished within 24 hours after BlackPearl received the last file or file part, BlackPearl will “truncate” the job and write the objects it did receive to the final storage target.
- When doing a GET/read job, clients should confirm that all bytes of every file have successfully been received by the client. The Helper classes do this confirmation automatically. If not all bytes were received, the .NET/C# SDK Helper class can identify this condition and automatically request the remaining bytes. For the Java SDK, the Helper class will detect when not all bytes are received and will throw an exception that the client can catch. If the client does not use the Helper class, the client needs to ensure that the length in the header of the GET operation matches the length of the actual data received.
User Cancellation of Archive Operation
Clients may allow users to cancel a BlackPearl archive/PUT operation while the operation is actively running. In this case, the client would typically want to stop the associated BlackPearl job and possibly delete the files that were already sent to BlackPearl.
The BlackPearl operation Cancel Job is used to cancel an active job. Any objects in the job that have been written in their entirety to physical data stores (tape or disk) are retained. Any objects in the job that were received in their entirety in cache are retained unless the optional “Force” flag is used.
If the Force flag is used, any objects in the job that have been partially written to physical data stores (regardless of whether or not they have been completely written to cache) are deleted and any space the object pieces consumed on physical data stores are marked as eligible for reclamation. BlackPearl does not expect any more objects from the job.
Note that even if the Force flag is set, the files that made it entirely to the physical data stores will not be deleted. Thus if a client wants to ensure that all files from a job are deleted from BlackPearl, after canceling the job it should issue a delete command for each of the files that still persist on BlackPearl.
The SDKs include commands to cancel a job. For example, in the Java SDK, the CancelJobSpectraS3Request class is used to cancel a job. This class also includes an optional withForce() method, which is used to set the force flag as described above.
Client Application Quits Unexpectedly
Developers should build their BlackPearl clients to be able to handle the client itself has crashing or quitting unexpectedly. Crashing may occur during an interaction with BlackPearl, such as during an archive or restore operation. When the client application has restarted, it should be able to identify at what point in the operation the client was during the crash and be able to properly continue or restart the operation.
Other Unexpected Conditions
BlackPearl clients may experience other unexpected conditions not listed above. Our BlackPearl Client Certification Program includes a questionnaire and a test plan that may help developers to consider other unexpected conditions. We recommend that developers review these documents and have their client certified to help ensure that their client is a robust and reliable as possible.