Restoring Large Files in Windows with BlackPearl – Sparse Files

Written by

Spectra Logic provides Software Development Kits (SDKs) for BlackPearl in several different programming languages to make it easier for our partners and customers to create integrations to BlackPearl. Some of the BlackPearl SDKs are used in Windows environments, particularly the Java and .NET SDKs. An issue was recently discovered in the way that these SDKs write large data blocks (64GB+) restored from BlackPearl on to disk.

In Windows, by default, when a file is created, Windows will initially reserve the space for the file on disk by filling the file with zeros. It then later replaces those zeros with the actual file byte content as it is received. When the files are 64GB or larger, it can take a very long time to write all of the zeros. In that time the file transfer from BlackPearl can time out, causing the SDK to have to resubmit its request to BlackPearl to restore the file. This resubmission causes the overall transfer process to take longer than it would if the resubmission did not occur. And in extreme cases with extremely large files (1TB+), the transfer can permanently fail.

To avoid this time out issue, the zeros should not be initially written in the file. This can be done by specifying that Windows Sparse Files be used. With Windows sparse files, per Microsoft, “the system does not allocate hard disk drive space to a file except in regions where it contains nonzero data”. Sparse files are therefore initially written much more quickly than non-sparse files.

We have updated our Java and .NET SDKs to use sparse files on restores. Partners and customers using these SDKs should update to the latest versions to ensure that they have this sparse file fix. However, the fixes in these SDKs will only be helpful if the “FileHelpers” class is being used, in which case the SDKs will be writing the restored files. For those organizations that are using these SDKs but have written their own code to write the restored files, they must implement sparse files themselves in their own code.

You can see examples of how Spectra Logic implemented sparse files in the SDKs:

Please Contact the Spectra Developer Program Team  if you need assistance with implementing sparse files with BlackPearl.