The open source Java Command-Line Interface (CLI) allows users to manage files in BlackPearl using a simple command-line interface. It can be downloaded from the Java CLI page on GitHub. Inside of the release download there is a bin directory and a lib directory. The bin directory contains all the executable files for both Linux and Windows. The lib directory contains all the jar files that are needed for the cli. There should be no need to modify anything inside of the lib directory. The only external dependency is the latest Java 8.
Command Prefix
Use ds3_java_cli as the prefix to call the CLI.
Example:
>ds3_java_cli -h
Be sure that the current directory contains the ds3_java_cli and ds3_java_cli.bat files.
Command Options
Note: the Java CLI is constantly being improved and enhanced. The information listed below may not be up to date. Use the -h (or -help) option to get the latest list of commands and command options.
-a <accessKeyId>
Access Key ID or have “DS3_ACCESS_KEY” set as an environment variable
-b <bucket>
The bucket to copy to
-bs <bufferSize>
Set the buffer size in bytes
-c <command>
The Command to execute. See “Commands” section below for details about each command.
--completed
Used with the command get_jobs to include the display of completed jobs
-d <directory>
Specify a directory to interact with if required
--debug
Debug output. If set takes precedence over the “verbose” option.
--discard
Discard restoration data (/dev/null) in get_bulk
-e <endpoint>
The BlackPearl endpoint to connect to or have “DS3_ENDPOINT” set as an environment variable. Use port 8080 (e.g. 192.168.56.101:8080) for the BlackPearl simulator.
--eject-label <arg>
Enter information to assist in the handling of the tapes.
--eject-location <arg>
Enter information to describe where the ejected tapes can be located.
--filter-params <arg>
Filter returned objects using the format key:value,key2:value2 (e.g. newerthan:d2.h3.m4.s5,larger than:100). Legal values:
contains -- specify full or partial object name, e.g. contains:projectX
owner -- specify object owner
newerthan, olderthan -- specify relative date from now in format d1.h2.m3.s4 (zero
values can be omitted , separate with ‘.’)
before, after -- specify absolute UTC date in format Y2016.M11.D9.h12.ZPDT
(zero values or UTC time zone can be omitted , separate with ‘.’)
largerthan, smallerthan -- specify object size in bytes
For example “--filter-params olderthan:D10”
--follow-symlinks
For Unix/Linux systems. On a PUT operation, if a symbolic link is discovered, the directories/files to which the symbolic link points will be included in the transfer.
--force
Used with the command “delete_bucket”. If this is set then the “delete_bucket” command will also delete all the objects in the bucket
-h,--help
Print Help Menu
--http
Send all requests over standard http
-i <id>
The ID for identifying api resources
--ignore-errors
Ignore files that cause errors
--ignore-naming-conflicts
Set true to ignore existing files of the same name and size during a bulk put
--in-cache
Set to filter out items that are only in cache. Used with get_suspect_objects
--insecure
Ignore ssl certificate verification
-k <secretKey>
Secret access key or have “DS3_SECRET_KEY” set as an environment variable.
--log-verbose
Log output to log file.
--log-debug
Debug (more verbose) output to log file.
--log-trace
Trace (most verbose) output to log file.
--metadata <arg>
Metadata for when putting a single object. Using the format: key:value,key2:value2
--modify-params <arg>
Parameters for modifying features using the format key:value,key2,value2. For modify_user
-n <numberOfFiles>
The number of files for the performance test. Used with the “performance” command
--no-follow-simlinks
Set to not follow symlinks, this is the default behavior
-nt <numberOfThreads>
Set the number of threads
-o <objectFileName>
The name of the object to be retrieved or stored
--output-format <arg>
Configure how the output should be displayed. Possible values: cli, json, csv
-p <prefix>
For “get_bulk”, restores only objects whose names start with the prefix. For “put_bulk”, appends the prefix to all objects. For “get_bucket”, lists only objects with the prefix
--priority <priority>
Set the bulk job priority. Possible values: critical, urgent, high, normal, low, background, minimized_due_to_too_many_retries
-r <retries>
Specifies how many times puts and gets will be attempted before failing the request. The default is 5
-s <sizeOfFiles>
The size (in MB) for each file in the performance test. Used with the “performance” command
--sync
Copy only the newest files. This feature is dependent on versioning and ObjectID being configured on the target bucket so that new version of the file is created. If versioning and ObjectID are not used we recommend using -p <prefix> to upload the new version to a new directory, or delete the old version before re-uploading.
-sv, --show-versions
Show version information
--trace
Trace output
--verbose
Verbose output
--verify-last-percent <percent>
Set verify last percent as an integer. Used with modify_data_path
-version
The Version_id of the object. Used for deleting a specific object version in the delete_object command
--writeOptimization <writeOptimization>
Set the job write optimization. Possible values: capacity, performance
-x <proxy>
The URL of the proxy server to use or have “http_proxy” set as an environment variable
Commands
get_bucket
Get a list of objects in the specified bucket.
get_bucket_details
Get information about a specific bucket.
get_bulk
Get the contents of one directory from BlackPearl.
get_cache_state
Gets the utilization information for all cache filesystems.
get_capacity_summary
Get the BlackPearl system-wide capacity summary.
get_config_summary
Get configuration information.
get_data_policy
Get information about the specified data policy
get_data_policies
Get information about all data policies
get_data_path_backend
Get information about the data path back-end.
get_detailed_objects
Filter an object list by size or creation date. Returns one line for each object. Optional ‘-b’ bucket_name
Optional ‘--filter-params’ to filter results. Use key:value pair key:value,key2:value2:… Legal values: newerthan, olderthan specify date from now in format d1.h2.m3.s4 (zero values can be omitted , separate with ‘.’); largerthan, smallerthan object size in bytes; contains (full or partial object name); owner (object owner)
get_detailed_objects_physical
Get a list of objects on tape, filtered by the size or creation date. Returns one line for each instance on the tape.
Optional ‘-b’ bucket_name. Optional ‘--filter-params’ to filter results. Use key:value pair key:value,key2:value2:… Legal values: newerthan, olderthan specify date from now in format d1.h2.m3.s4 (zero values can be omitted, separate with ‘.’); largerthan, smallerthan object size in bytes; contains (full or partial object name); owner (object owner)
get_job
Get information about one job.
get_jobs
Get a list of all currently-running jobs.
get_object
Retrieve one object from the BlackPearl.
get_objects_on_tape
Get the list of object parts on the specified tape
get_physical_placement
Get the location(s) of the specified object in the BlackPearl storage environment. Especially useful for objects that might be on an ejected tape.
get_service
Get a list of buckets on BlackPearl.
get_storage_domain
Get information about all storage domains.
Optional -i (UUID or name) restricts output to one storage domain.
Optional --writeOptimization (capacity | performance) filters results to those matching write optimization.
get_tapes
Get a list of all tapes.
get_tape_failure
Get a list of all tape failures
get_user
Get information about the specified user
get_users
Get information about all users
eject_tape
Ejects a single tape. Provide tape ID or barcode using -i option.
Tapes are not eligible for ejection if mediaEjectionAllowed=FALSE for the storage domain.
If tape is being used for a job, it is ejected once it is no longer in use.
eject_storage_domain
Ejects all eligible tapes within the specified storage domain.
Tapes are not eligible for ejection if mediaEjectionAllowed=FALSE for the storage domain.
If tape is being used for a job, it is ejected once it is no longer in use.
system_information
Get general BlackPearl system information.
head_object
Get object metadata information
put_bucket
Create a new bucket.
put_bulk
Put the contents of one directory on to BlackPearl.
put_job
Modify the priority of a job that is in process.
put_object
Put one file on to the BlackPearl.
delete_bucket
Delete a bucket.
delete_folder
Delete a folder.
delete_job
Delete the specified job.
delete_object
Delete an object on BlackPearl.
delete_tape
Delete one tape from BlackPearl.
delete_tape_drive
Remove the specified offline tape drive.
delete_tape_partition
Remove the specified offline tape partition.
delete_tape_failure
Deletes the specified tape failure
performance
Run performance test to benchmark the transfer rate to a Spectra S3 endpoint.
modify_data_policy
Alter parameters for the specified data policy. Requires the ‘-i’ parameter to specify data policy (UUID or name). Requires the ‘--modify-params’ parameter to be set.Use key:value pair key:value,key2:value2:… Legal values: name, checksum_type, default_blob_size, default_get_job_priority, default_put_job_priority, default_verify_job_priority, rebuild_priority, end_to_end_crc_required, versioning. Use the get_data_policies command to retrieve a list of policies and current values.
modify_user
Modify the name or default data policy for a user
reclaim_cache
Forces a full reclaim of all caches, and waits until the reclaim completes. Cache contents that need to be retained because they are a part of an active job are retained. Any cache contents that can be reclaimed will be. This operation may take a very long time to complete, depending on how much of the cache can be reclaimed and how many blobs the cache is managing.
verify_bulk_job
Create a job to verify objects. A VERIFY job reads data from the permanent data store and verifies that the CRC of the data read matches the expected CRC. VERIFY jobs ALWAYS read from the data store -- even if the data currently resides in cache
verify_tape
Verifies the media content on the specified tape.
verify_all_tapes
Verify the integrity of all tapes in BlackPearl.
verify_pool
Force a verification to be scheduled immediately for the specified pool, without having to wait for the automatic, periodic scheduling.
verify_all_pools
Force a verification to be scheduled immediately for the all pools, without having to wait for the automatic, periodic scheduling.
verify_system_health
Verifies that the system appears to be online and functioning normally, and that there is adequate free space for the database file system.
Environment Variables
You can use environment variables to more easily set some of the command options.
On Linux you can create a RC or resource file that you use to store common configurations without having to specify those arguments from the command line each time the CLI is executed. The following is an example of a resource file in Linux:
export DS3_ACCESS_KEY=“access_key“
export DS3_SECRET_KEY=“secret_key“
export DS3_ENDPOINT=“hostname:8080“
To use the RC file use source my_rc_file.rc which will export all of the environment variables into the current bash shell and will be picked up by the CLI. The help menu describes all the arguments that can be specified from the command line.
On Windows you can create a resource BAT file to do the same thing. Here is the same example from above, but as a BAT file:
set DS3_ACCESS_KEY=access_key
set DS3_SECRET_KEY=secret_key
set DS3_ENDPOINT=hostname:8080
To use the BAT file just run it from the Windows CLI and the ds3_java_cli.bat script will be able to use the values. Note: ds3_java_cli.bat has to be run from the same Windows CLI that the resource BAT file was run in.
The CLI supports connecting to BlackPearl via a HTTP Proxy. To automatically connect to the proxy from the CLI set http_proxy as an environment variable and the ds3_java_cli will pick up the configuration. The proxy setting it not required to be set, but if you work in an environment where a proxy is present and the http_proxy environment variable is already set, you should not have to do anything.
Command Examples
1. Create a new bucket called “bucket411”. Use http instead of https. Use the BlackPearl simulator (on port 8080).
ds3_java_cli -e 192.168.56.101:8080 -a amVmZmJy -k iYkqUwcd -c put_bucket -b bucket411 --http
2. Put a directory called “subdir1” in bucket “jeffbr1”. Use http instead of https.
ds3_java_cli -e bp1.spectralogic.com -a amVmZmJy -k dUwVDkj6 -c put_bulk -b jeffbr1 -d “c:\Temp\putithere\subdir1” --http
Remember that you can optionally set environment variables for the S3 Access ID (-a), S3 Secret Key (-k), endpoint (-e) and HTTP proxy (-x, not shown). The environment variables are explained in the section above. Also note that the S3 Access ID and Secret Key provided in the examples above are a sample only, and these specific values will not work in your setup. You can learn how to create users and generate their Access ID and Secret Key by reading the Simulator Installation and Usage Instructions.
More examples can be found in the Java CLI Samples zip file. The samples files were created for a Windows system (though the commands will work on any system), so all files end with a .BAT file extension.
Zip and Archive Example
A common use case for the Java CLI might be to have a cron job or scheduled task run on some periodic basis to move files from a server folder to BlackPearl. In some cases it might be beneficial to first put the files in a ZIP or TAR file before moving them to BlackPearl. We have created an example file set showing a sample of zipping and archiving a file to BlackPearl. The sample can be found in the Zip and Archive Sample zip file.
This sample includes a BAT file with two commands.
The first command creates the zip file using the open source 7zip (7zip.org). The zip file name includes the date that the file was created.
7za a -tzip “Archives\Dailys\VOD_%DATE:~7,2%.%DATE:~4,2%.%DATE:~-4%.zip” “Projects\Deliverables\VOD\*.*” -r -mx5
The second command uses the Java CLI to move the generated zip file to BlackPearl (this uses port 8080 which is the BlackPearl simulator).
ds3_java_cli.bat -e 192.168.56.102:8080 -a amVmZmJy -k iYkqUwcd -c put_object -b bucketJeff1 -o “Archives\Dailys\VOD_%DATE:~7,2%.%DATE:~4,2%.%DATE:~-4%.zip” --http --verbose
For additional assistance with the Java CLI please use our Forums.