Java CLI Reference & Examples

The open source Java Command-Line Interface (CLI) allows users to manage files in BlackPearl using a simple command-line interface. It can be downloaded from the Java CLI page on GitHub. Inside of the release download there is a bin directory and a lib directory. The bin directory contains all the executable files for both Linux and Windows. The lib directory contains all the jar files that are needed for the cli. There should be no need to modify anything inside of the lib directory. The only external dependency is the latest Java 7 or 8. For Windows we recommend installing the Java Runtime Edition (JRE). For Mac we recommend installing the Java Development Kit (JDK).

Command Prefix

Use ds3_java_cli as the prefix to call the CLI.

Example:

>ds3_java_cli -h

Be sure that the current directory contains the ds3_java_cli and ds3_java_cli.bat files.

Command Options

Note: the Java CLI is constantly being improved and enhanced. The information listed below may not be up to date. Use the -h (or -help) option to get the latest list of commands and command options.

-a <accessKeyId>                            

Access Key ID or have “DS3_ACCESS_KEY” set as an environment variable

-b <bucket>                                

The bucket to copy to

-bs <bufferSize>

Set the buffer size in bytes

-c <command>                                

The Command to execute. See “Commands” section below for details about each command.

--checksum

Validate checksum values.

--completed

Used with the command get_jobs to include the display of completed jobs

-d <directory>                              

Specify a directory to interact with if required

--debug

Debug output. If set takes precedence over the “verbose” option.

--discard

Discard restoration data (/dev/null) in get_bulk

-e <endpoint>                              

The BlackPearl endpoint to connect to or have “DS3_ENDPOINT” set as an environment variable. Use port 8080 (e.g. 192.168.56.101:8080) for the BlackPearl simulator.

--eject-label <arg> 

Enter information to assist in the handling of the tapes.

--eject-location <arg>

Enter information to describe where the ejected tapes can be located.

--filter-params <arg>

Filter returned objects using the format key:value,key2:value2 (e.g. newerthan:d2.h3.m4.s5,larger than:100). Legal values:

contains -- specify full or partial object name, e.g. contains:projectX

owner -- specify object owner

newerthan, olderthan -- specify relative date from now in format d1.h2.m3.s4 (zero

values can be omitted , separate with ‘.’)

before, after -- specify absolute UTC date in format Y2016.M11.D9.h12.ZPDT

(zero values or UTC time zone can be omitted , separate with ‘.’)

largerthan, smallerthan -- specify object size in bytes

For example “--filter-params olderthan:D10”

--follow-symlinks

For Unix/Linux systems. On a PUT operation, if a symbolic link is discovered, the directories/files to which the symbolic link points will be included in the transfer.

--force                                  

Used with the command “delete_bucket”. If this is set then the “delete_bucket” command will also delete all the objects in the bucket

-h,--help                                  

Print Help Menu

--http                                  

Send all requests over standard http

-i <id>                                    

The ID for identifying api resources

--ignore-errors

Ignore files that cause errors

--ignore-naming-conflicts

Set true to ignore existing files of the same name and size during a bulk put

--in-cache

Set to filter out items that are only in cache.  Used with get_suspect_objects

--insecure                              

Ignore ssl certificate verification

-k <secretKey>                              

Secret access key or have “DS3_SECRET_KEY” set as an environment variable.

--log-verbose

Log output to log file.

--log-debug

Debug (more verbose) output to log file.

--log-trace

Trace (most verbose) output to log file.

--metadata <arg>

Metadata for when putting a single object. Using the format: key:value,key2:value2

--modify-params <arg>

Parameters for modifying features using the format key:value,key2,value2. For modify_user

-n <numberOfFiles>             

The number of files for the performance test. Used with the “performance” command

--no-follow-simlinks

Set to not follow symlinks, this is the default behavior

-nt <numberOfThreads>   

Set the number of threads

-o <objectFileName>                        

The name of the object to be retrieved or stored

--output-format <arg>                    

Configure how the output should be displayed. Possible values: cli, json, csv

-p <prefix>                                

For “get_bulk”, restores only objects whose names start with the prefix. For “put_bulk”, appends the prefix to all objects. For “get_bucket”, lists only objects with the prefix

--priority <priority>                    

Set the bulk job priority. Possible values: critical, urgent, high, normal, low, background, minimized_due_to_too_many_retries

-r <retries>                                

Specifies how many times puts and gets will be attempted before failing the request. The default is 5

 -s <sizeOfFiles>  

The size (in MB) for each file in the performance test. Used with the “performance” command

--sync        

Copy only the newest files

-sv, --show-versions

Show version information

--trace  

Trace output

--verbose                                

Verbose output

--verify-last-percent <percent>

Set verify last percent as an integer.  Used with modify_data_path

--writeOptimization <writeOptimization>  

Set the job write optimization. Possible values: capacity, performance

-x <proxy>                                  

The URL of the proxy server to use or have “http_proxy” set as an environment variable

Commands

get_bucket

Get a list of objects in the specified bucket.

get_bulk

Get the contents of one directory from BlackPearl.

get_cache_state

Gets the utilization information for all cache filesystems.

get_capacity_summary

Get the BlackPearl system-wide capacity summary.

get_config_summary

Get configuration information.

get_data_policy

Get information about the specified data policy

get_data_policies

Get information about all data policies

get_data_path_backend

Get information about the data path back-end.

get_detailed_objects

Filter an object list by size or creation date. Returns one line for each object. Optional ‘-b’ bucket_name

Optional ‘--filter-params’ to filter results. Use key:value pair key:value,key2:value2:… Legal values: newerthan, olderthan specify date from now in format d1.h2.m3.s4 (zero values can be omitted , separate with ‘.’); largerthan, smallerthan object size in bytes; contains (full or partial object name); owner (object owner)

get_detailed_objects_physical

Get a list of objects on tape, filtered by the size or creation date. Returns one line for each instance on the tape.

Optional ‘-b’ bucket_name. Optional ‘--filter-params’ to filter results. Use key:value pair key:value,key2:value2:… Legal values: newerthan, olderthan specify date from now in format d1.h2.m3.s4 (zero values can be omitted, separate with ‘.’); largerthan, smallerthan object size in bytes; contains (full or partial object name); owner (object owner)

get_job

Get information about one job.

get_jobs

Get a list of all currently-running jobs.

get_object

Retrieve one object from the BlackPearl.

get_objects_on_tape

Get the list of object parts on the specified tape

get_physical_placement

Get the location(s) of the specified object in the BlackPearl storage environment. Especially useful for objects that might be on an ejected tape.

get_service

Get a list of buckets on BlackPearl.

get_storage_domain

Get information about all storage domains.

Optional -i (UUID or name) restricts output to one storage domain.

Optional --writeOptimization (capacity | performance) filters results to those matching write optimization.

get_tapes

Get a list of all tapes.

get_tape_failure

Get a list of all tape failures

get_user

Get information about the specified user

get_users

Get information about all users

eject_tape

Ejects a single tape. Provide tape ID or barcode using -i option.

Tapes are not eligible for ejection if mediaEjectionAllowed=FALSE for the storage domain.

If tape is being used for a job, it is ejected once it is no longer in use.

eject_storage_domain

Ejects all eligible tapes within the specified storage domain.

Tapes are not eligible for ejection if mediaEjectionAllowed=FALSE for the storage domain.

If tape is being used for a job, it is ejected once it is no longer in use.

system_information

Get general BlackPearl system information.

head_object

Get object metadata information

put_bucket

Create a new bucket.

put_bulk

Put the contents of one directory on to BlackPearl.

put_job

Modify the priority of a job that is in process.

put_object

Put one file on to the BlackPearl.

delete_bucket

Delete a bucket.

delete_folder

Delete a folder.

delete_job

Delete the specified job.

delete_object

Delete an object on BlackPearl.

delete_tape

Delete one tape from BlackPearl.

delete_tape_drive

Remove the specified offline tape drive.

delete_tape_partition

Remove the specified offline tape partition.

delete_tape_failure

Deletes the specified tape failure

performance

Run performance test to benchmark the transfer rate to a Spectra S3 endpoint.

modify_data_policy

Alter parameters for the specified data policy. Requires the ‘-i’ parameter to specify data policy (UUID or name). Requires the ‘--modify-params’ parameter to be set.Use key:value pair key:value,key2:value2:… Legal values: name, checksum_type, default_blob_size, default_get_job_priority, default_put_job_priority, default_verify_job_priority, rebuild_priority, end_to_end_crc_required, versioning. Use the get_data_policies command to retrieve a list of policies and current values.

modify_user

Modify the name or default data policy for a user

reclaim_cache

Forces a full reclaim of all caches, and waits until the reclaim completes. Cache contents that need to be retained because they are a part of an active job are retained. Any cache contents that can be reclaimed will be. This operation may take a very long time to complete, depending on how much of the cache can be reclaimed and how many blobs the cache is managing.

verify_bulk_job

Create a job to verify objects. A VERIFY job reads data from the permanent data store and verifies that the CRC of the data read matches the expected CRC. VERIFY jobs ALWAYS read from the data store -- even if the data currently resides in cache

verify_tape

Verifies the media content on the specified tape.

verify_all_tapes

Verify the integrity of all tapes in BlackPearl.

verify_pool

Force a verification to be scheduled immediately for the specified pool, without having to wait for the automatic, periodic scheduling.

verify_all_pools

Force a verification to be scheduled immediately for the all pools, without having to wait for the automatic, periodic scheduling.

verify_system_health

Verifies that the system appears to be online and functioning normally, and that there is adequate free space for the database file system.

Environment Variables

You can use environment variables to more easily set some of the command options.

On Linux you can create a RC or resource file that you use to store common configurations without having to specify those arguments from the command line each time the CLI is executed. The following is an example of a resource file in Linux:

export DS3_ACCESS_KEY=access_key
export
DS3_SECRET_KEY=secret_key
export
DS3_ENDPOINT=hostname:8080

To use the RC file use source my_rc_file.rc which will export all of the environment variables into the current bash shell and will be picked up by the CLI. The help menu describes all the arguments that can be specified from the command line.

On Windows you can create a resource BAT file to do the same thing. Here is the same example from above, but as a BAT file:

set DS3_ACCESS_KEY=access_key
set DS3_SECRET_KEY=secret_key
set DS3_ENDPOINT=hostname:8080

To use the BAT file just run it from the Windows CLI and the ds3_java_cli.bat script will be able to use the values. Note: ds3_java_cli.bat has to be run from the same Windows CLI that the resource BAT file was run in.

The CLI supports connecting to BlackPearl via a HTTP Proxy. To automatically connect to the proxy from the CLI set http_proxy as an environment variable and the ds3_java_cli will pick up the configuration. The proxy setting it not required to be set, but if you work in an environment where a proxy is present and the http_proxy environment variable is already set, you should not have to do anything.

Command Examples

1. Create a new bucket called “bucket411”. Use http instead of https. Use the BlackPearl simulator (on port 8080).

ds3_java_cli -e 192.168.56.101:8080 -a amVmZmJy -k iYkqUwcd -c put_bucket -b bucket411 --http

2. Put a directory called “subdir1” in bucket “jeffbr1”. Use http instead of https.

ds3_java_cli -e bp1.spectralogic.com -a amVmZmJy -k dUwVDkj6 -c put_bulk -b jeffbr1 -d “c:\Temp\putithere\subdir1” --http

Remember that you can optionally set environment variables for the S3 Access ID (-a), S3 Secret Key (-k), endpoint (-e) and HTTP proxy (-x, not shown). The environment variables are explained in the section above. Also note that the S3 Access ID and Secret Key provided in the examples above are a sample only, and these specific values will not work in your setup. You can learn how to create users and generate their Access ID and Secret Key by reading the Simulator Installation and Usage Instructions.

More examples can be found in the Java CLI Samples zip file. The samples files were created for a Windows system (though the commands will work on any system), so all files end with a .BAT file extension.

Zip and Archive Example

A common use case for the Java CLI might be to have a cron job or scheduled task run on some periodic basis to move files from a server folder to BlackPearl. In some cases it might be beneficial to first put the files in a ZIP or TAR file before moving them to BlackPearl. We have created an example file set showing a sample of zipping and archiving a file to BlackPearl. The sample can be found in the Zip and Archive Sample zip file.

This sample includes a BAT file with two commands.

The first command creates the zip file using the open source 7zip (7zip.org). The zip file name includes the date that the file was created.

7za a -tzip “Archives\Dailys\VOD_%DATE:~7,2%.%DATE:~4,2%.%DATE:~-4%.zip” “Projects\Deliverables\VOD\*.*” -r -mx5

The second command uses the Java CLI to move the generated zip file to BlackPearl (this uses port 8080 which is the BlackPearl simulator).

ds3_java_cli.bat -e 192.168.56.102:8080 -a amVmZmJy -k iYkqUwcd -c put_object -b bucketJeff1 -o “Archives\Dailys\VOD_%DATE:~7,2%.%DATE:~4,2%.%DATE:~-4%.zip”  --http --verbose

For additional assistance with the Java CLI please use our Forums.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>