Note that all output files from a job, including task outputs, are stored in the same container. This means that storage throttling limits may be enforced if a large number of tasks try to persist files at the same time.

Listing and Retrieving Files in Client Code

To access persisted files from client code, you must configure the client with the details of the linked storage account. Then use the JobOutputStorage and TaskOutputStorage constructors that take a CloudStorageAccount, or the extension methods on CloudJob and CloudTask.

var job = await batchClient.JobOperations.GetJobAsync(jobId);
var jobOutputStorage = job.OutputStorage(linkedStorageAccount);

var jobOutputBlob = jobOutputStorage.ListOutputs(JobOutputKind.JobOutput)
                                    .SingleOrDefault()
                                    as CloudBlockBlob;

if (jobOutputBlob != null)
{
    await jobOutputBlob.DownloadToFileAsync("movie.mp4", FileMode.Create);
}

Conventions

The conventions library defines paths in Azure blob storage for output storage. All outputs from a job, including task outputs, are stored in a single container. Within that container, outputs are stored by kind and (for task outputs) task ID. This section describes the conventions for the job output container name and for paths within the job output container.

Job Output Container Name

The job output container name is formed according to the following rules:

• Normalize the job ID to lower case. (Due to the restricted set of letters permitted in IDs, there are no locale issues with this normalization.)
• If prepending "job-" to the normalized ID gives a valid container name, use that.
• Otherwise:
  • Calculate the SHA1 hash of the normalized ID, and express it as a 40-character hex string.
  • Replace all underscores, colons, and sequences of one or more hyphens in the normalized ID by single hyphens, then remove any leading or trailing hyphens.
  • If the resulting string is empty, use the string "job" instead.
  • If the resulting string is longer than 15 characters, truncate it to 15 characters. If truncation results in a trailing hyphen, remove it.
  • The container name is the string "job-", followed by the truncated ID, followed by a hyphen, followed by the hash.

For example, if the job ID is MyTerrificJob, then the container name is job-myterrificjob as this is a valid container name. If the job ID is my-_EVEN_MORE_-terrific-job, we cannot use job-my-_even_more_-terrific-job as this is not a valid container name, so we apply the algorithm:

• The SHA1 hash of my-_even_more_-terrific-job (all lower case) is 68b05a7d8aa6aa65b9a6892c667a6c406a16ad65.
• Replacing hyphens and underscores by single hyphens in the lower case ID gives my-even-more-terrific-job. There are no leading or trailing hyphens to remove.
• Truncating to 15 characters gives us my-even-more-te. Again there are no leading or trailing hyphens to remove.
• The final container name is job-my-even-more-te-68b05a7d8aa6aa65b9a6892c667a6c406a16ad65.

The purpose behind this algorithm is to ensure that jobs are given valid and unique container names, while preserving human readability as far as possible, by where possible using the job ID, and in other cases including a prefix based on the job ID.

Blob Path

The blob path within the container depends on whether the output is being stored as a job output or task output.

Job outputs are stored as "${kind}/{filename}". For example, if the file "out/mergeresults.txt" is stored under JobOutputKind.JobOutput, then its path within the container is "$JobOutput/out/mergeresults.txt".

Task outputs are stored as "{taskid}/${kind}/{filename}". For example, if the file "analytics.log" from task "analysis-309" is stored under TaskOutputKind.TaskLog, then its path within the container is "analysis-309/$TaskLog/analytics.log".

The purpose behind this structure is to enable clients to readily locate outputs based on their kind - for example, "list the main outputs of the job" or "list the log files for task analysis-309".