Rio Broker Best Practices

Spectra Logic’s Rio Broker is one method for integrating to BlackPearl. While the REST API of Rio Broker is relatively straightforward, developers must thoughtfully build their Rio Broker integration to ensure that the integration is complete, reliable, and fast. The following Rio Broker design best practices should be followed to ensure that the integration is successful and can pass Spectra’s Certification Program.

Below we will refer to the software that integrates with Rio Broker as the “client”.

Maximizing Transfer Rate

While Rio Broker is responsible for moving the files to be archived or restored to BlackPearl, the client also plays an important part in ensuring fast file transfers. There are two areas where the client can contribute to maximize transfer performance:

  • Include as many files into one command as possible. When issuing an individual archive or restore command, the client specifies one or more files to be transferred. Every effort should be made to include as many files as possible into each individual archive or restore command. This will allow Rio Broker to group the files when interacting with BlackPearl, which works faster and more efficiently with groups of files. Please note though that Rio Broker currently has a limit of 1,000 files per job, so do not specify more than 1,000 files in each archive or restore command.
  • Send transfer requests in parallel. Rio Broker can process multiple archive and restore commands in parallel. Clients should send the archive and restore requests as soon as possible and in parallel to achieve maximum Rio Broker transfer performance.

Jobs Management

In Rio Broker, an archive or restore operation is called a “Job”. The status of an actively-running job can be determined with the following command: GET /api/jobs/{jobId}.  The job should be monitored by the client until the status of the job changes to “completed”.

Many clients will have a need to cancel actively running Rio Broker jobs. This can be done using the following command: PUT /api/jobs/{jobId}. Be sure to set the “cancel” parameter to true. When canceling a Rio Broker job, it is up to the client to determine which files have already been transferred in the archive or restore operation and then clean them up accordingly. When Rio Broker receives a cancel command, it will cease further transfers, but it will not remove any files that were already transferred.

Session Management

In order to send API commands to Rio Broker, the client must include a session authentication token. The authentication token can be acquired using the following command: POST /api/tokens. The client will also need to pass in the username and password parameters (spectra/spectra). The authentication token will expire in one hour, so the client will need to track the age of the token and request a new one every hour.

Use .NET SDK

Spectra Logic has provided a Rio Broker .NET SDK for clients that are written in .NET. This SDK makes client integration easier and faster. We recommend using this SDK if at all possible. See our Documentation page for help on using this SDK.

Managing Unexpected Conditions

Job Failure Error

The status of a Rio Broker transfer job (archive and restore) can be found using the following API command: GET /api/jobs/{jobId}. The job status returned will be one of the following: active, completed, paused, canceled, error. If the job status is “error”, the job has failed for some reason. The client needs to be able to handle an error status appropriately. Here are some of the reasons that an error could occur:

  • Files that the client is attempting to restore are only available on ejected tape (requires operator to insert tapes back into tape library)
  • BlackPearl is unavailable
  • BlackPearl storage target (tape, disk, cloud, replicated BlackPearl) is unavailable
  • Source storage system is unavailable
  • BlackPearl cache is full
  • Restore is taking too long
  • Rio Broker does not have permissions to files on BlackPearl (bad S3 credentials)
  • RioBroker doesn’t have permissions to files on source storage (bad FTP or file system credentials)
  • File does not exist and therefore cannot be moved
  • File already exists and therefore cannot be overwritten

Clients should be designed to handle the error status and display the status message to the user. In many cases the message will include important information for the user. For example, if the files are only available on ejected tapes, the message will include the tape barcodes so that the user can find the tapes and re-insert them into the tape library.

Rio Broker Unavailable

It is possible that your client may not be able to communicate with Rio Broker due to the network or the Rio Broker server being down. In this case, the client must handle this condition and must provide a useful message to the user so that they can attempt to resolve the issue.

Partial File Restore

Rio Broker includes a feature called Time-Based Partial File Restore (PFR). Using this feature a client can restore part of a video file by specifying the time range.

In order to use PFR, the client must set the parameter “indexMedia” to “true” when the file is archived. This indexing will allow the PFR feature to be used on the file.

When the file is restored, the client must specify the “timeCodeRange” parameter with the time range of the file that is desired.

There are two possible errors that could occur when using PFR:

  • When attempting to archive the file, it is possible that the file cannot be indexed.
  • When attempting to restore the partial file, it is possible that the partial file cannot be created.

In both of these cases, the job status will be “error”. The client should be able to handle these errors.

Maintaining State

It is possible that the client, independent of Rio Broker, could crash, or the server on which the client is running could reboot. In this case, the client will need to be able to return to its previous state and know the status of all of the active Rio Broker jobs. The Rio Broker command GET /api/jobs can be used to get the status of all Rio Broker jobs.

Including File Size on Archive

When issuing an archive command (POST /api/brokers/{brokerName}/archive), one of the input parameters is the size of each file to be transferred in bytes. This file size is a required parameter when the source storage has an FTP interface. For source storage with a file path interface (e.g. \\server1\shareA), the file size parameter is optional -- Rio Broker can determine the file size on its own. However, if at all possible, the client should provide the file size to maximize Rio Broker performance.

Importing Foreign LTFS Tapes

BlackPearl has the ability to import foreign LTFS tapes and make the files on those tapes visible to Rio Broker. In some cases your customers may want to import these foreign LTFS tapes and make them accessible in your client. New files can be identified in Rio Broker by using the command PUT /api/brokers/{brokerName}/agents/{agentName} and specifying the “index” and “re-index” parameters both as “true”. The client can then choose to add these new files to its catalog/database and possibly restore the files if appropriate.

Multiple Rio Broker Systems for Failover

BlackPearl supports the ability to replicate data to another BlackPearl, typically at another physical site. Replication is set at the bucket level in BlackPearl (a bucket in BlackPearl corresponds to a broker in Rio Broker), and this replication is typically set up to be bidirectional.

A Rio Broker server can be placed in front of each of these BlackPearls so that the customer has a fully redundant archive storage system at each site. In the scenario, the client will need to be aware of each Rio Broker server endpoint and be able to fail over to the second Rio Broker if the first Rio Broker is not available. In the event that the fail over occurs, the client may need to tell the secondary Rio Broker to update its file index with the command PUT /api/brokers/{brokerName}/agents/{agentName} and specifying the “index” and “re-index” parameters both as “true”.

Utilizing Online Documentation -- Swagger Docs

Once you have installed Rio Broker on a computer, the online API help documents can be accessed at https://localhost:5050/api/viewer/index.html. This swagger HTML version of the documentation is easier to use than the PDF version of the API documentation that is available on this site.

Installation and Operation Guide for Your Users

Make sure to provide your customers with written instructions for configuring and operating their Rio Broker client in your application. Customers should be able to easily figure out how to set up and use the client. Remember that Rio Broker requires that a cluster, broker, and agent be set up before it can be used. If your client doesn’t set up these objects itself, then you will need to explain in your written instructions how the customer does this themselves. Remember that the client is the customer’s primary interface not only to your application, but also to Rio Broker and BlackPearl.