{"id":4897,"date":"2022-04-26T12:06:49","date_gmt":"2022-04-26T12:06:49","guid":{"rendered":"https:\/\/petabytz.com\/?p=4897"},"modified":"2026-02-20T06:44:30","modified_gmt":"2026-02-20T06:44:30","slug":"transcoding-video-files-with-s3-batch-operations","status":"publish","type":"post","link":"https:\/\/petabytz.com\/blogs\/transcoding-video-files-with-s3-batch-operations\/","title":{"rendered":"Transcoding video files with S3 Batch Operations"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Transcoding video files with S3 Batch Operations<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Many media companies store their extensive video repositories in S3. Typical practice suggests storing current videos in S3 Standard and archiving older videos into S3 Glacier. Lambda invocations could be triggered on S3 events such as putting a new object into a bucket. S3 Batch Operations provides you with a managed solution to assist with triggering Lambda functions for existing objects and performing other large-scale tasks in S3.<\/p>

Create a Lambda transcoding workflow<\/strong><\/p>

First, set up the VOD automation workflow using the post linked above. When you reach step 2, creating the Lambda function, return to this post. You must change the Lambda function so it works with S3 Batch Operations. This modification in the Lambda function uses metadata from a new field in the event JSON passed to Lambda and sends a job response back to S3 Batch Operations.<\/p>

The VOD Automation post uses an S3 trigger in Lambda to point to the bucket ingesting video files. In the Lambda function you create for S3 Batch Operations, do not setup an S3 trigger as your S3 Batch Operations job will trigger the Lambda function directly.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In the convert.py file, add the following lines of code at the beginning to extract the S3 Batch Operations event ID and specific task information, such as S3Bucket and S3Key.<\/p>

First, the S3 Batch Operations job will send job parameters to Lambda in the event JSON, which includes\u00a0jobId<\/strong>,\u00a0invocationId<\/strong>, and\u00a0invocationSchemaVersion<\/strong>. Lambda grabs these parameters by adding the following three lines<\/p>

Python<\/p>

jobId = event[\u2018job\u2019][\u2018id\u2019]<\/p>

invocationId = event[\u2018invocationId\u2019] invocationSchemaVersion = event[\u2018invocationSchemaVersion\u2019]<\/p>

With the specific S3 bucket and key information now in the event JSON task dictionary, a few more lines extract this information directly in the event records. Add the following lines:<\/p>

Python<\/p>

task = event[\u2018tasks\u2019][0]<\/p>

taskId = task[\u2018taskId\u2019]<\/p>

sourceS3BucketArn = task[\u2018S3BucketArn\u2019]<\/p>

sourceS3Key = task[\u2018S3Key\u2019]<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Each invocation of Lambda from S3 Batch Operations also needs a results dictionary to include\u00a0taskId<\/strong>,\u00a0resultCode<\/strong>, and\u00a0resultString<\/strong>. I set\u00a0resultCode\u00a0<\/strong>and\u00a0resultString\u00a0<\/strong>depending on the response from the AWS Elemental MediaConvert jobs. For example, if there is an exception when submitting the MediaConvert jobs, then\u00a0resultCode\u00a0<\/strong>= 500 and\u00a0resultString\u00a0<\/strong>= e for the exception string. Then at the end of the MediaConvert job, I\u2019ve added the following code block to add results to the results array:<\/p>

Python<\/p>

finally:<\/p>

results.append({<\/p>

\u2018taskId\u2019: taskId,<\/p>

\u2018resultCode\u2019: resultCode,<\/p>

\u2018resultString\u2019:<\/p>

resultString,<\/p>

})<\/p>

When triggering Lambda with S3 Batch Operations, a new response needs to be created that returns special data fields. This tells S3 Batch Operations if the Lambda function successfully executed each task. In this specific case, Lambda returns whether or not the MediaConvert job was successfully submitted:<\/p>

Python<\/p>

return {<\/p>

\u2018invocationSchemaVersion\u2019: invocationSchemaVersion,<\/p>

\u2018treatMissingKeyAs\u2019: \u2018Permenant Failure\u2019,<\/p>

\u2018invocationId\u2019: invocationId,<\/p>

\u2018results\u2019: results<\/p>

}<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Create an S3 Batch Operations job using the console<\/h2>

In the S3 console and choose Buckets,\u00a0Batch Operations\u00a0<\/strong>on the left tab under Buckets.<\/p>

Choose\u00a0Create Job<\/strong>.<\/p>

Choose the appropriate Region for your S3 bucket. Under\u00a0Choose manifest<\/strong>, select where your objects are stored and the\u00a0CSV\u00a0<\/strong>file and enter the path to your S3 bucket. If your manifest contains version IDs, make sure to check that box, and choose\u00a0Next<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Select\u00a0Invoke AWS Lambda function<\/strong>, and select the Lambda function you created \u2014 in this case\u00a0VODLambdaConvertBatch<\/strong>. Select the function version if you need a different version from $LATEST.<\/p>

Choose\u00a0Next<\/strong>.<\/p>

In Step 3 of the setup wizard, give your job a description and priority level, and choose a report type and destination.<\/p>

Choose an IAM role for your S3 Batch Operations job to assume. To learn more about the necessary role and invoking Lambda functions in S3, Select a role with permissions to s3:GetObject and s3:GetObjectVersion for the object source buckets as well as the bucket that holds the manifest file.<\/p>

The role also needs s3:PutObject for the destination bucket for the job completion report.<\/p>

Lastly, the role needs the lambda:InvokeFunction permission for the Lambda function that it invokes. In this case, the role name is S3BatchLambdaRole.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Choose\u00a0Next<\/strong>.<\/p>

In Step 4, review and verify your job parameters before choosing\u00a0Create Job.<\/strong><\/p>

After S3 finishes reading your job\u2019s manifest, it moves the job to the\u00a0Awaiting your confirmation\u00a0<\/strong>From here, you can check the number of objects in the manifest and choose\u00a0Confirm the job\u00a0<\/strong>to run it.<\/p>

After the job starts running, you can check its object-level progress through the console dashboard view or by selecting the specific job. As each Lambda invocation occurs, S3 writes logs to CloudWatch Logs.<\/p>

In the Lambda console, select your Lambda function, and choose Monitoring. You can see Lambda Invoke metrics along with an easy navigation button to View logs in CloudWatch.<\/p>

When the S3 Batch Operations job completes, view the Successful and Failed object counts to confirm that everything performed as expected. For the details on failed objects, see your job report.<\/p>

To monitor video transcode jobs, navigate to the MediaConvert service to monitor transcoding jobs for each of the files.<\/p>

Changes for S3 Glacier Restore<\/h2>

When creating your Lambda function using the VOD Automation workflow, configure an S3 trigger on the S3 bucket that receives your restored videos. For the event type, choose\u00a0Restore from Glacier Completed<\/strong>. Create a separate Lambda function for triggering with S3, as the function that you created earlier needed specific additions for S3 Batch Operations reporting.<\/p>

Create the S3 Batch Operations job as before, with the following changes:<\/p>

In step 2, instead of selecting\u00a0Invoke AWS Lambda function<\/strong>, select\u00a0Restore.<\/strong><\/p>

Under\u00a0Restore options<\/strong>, select the number of days that the restored objects should remain available. The original objects remain in S3 Glacier. After the selected number of days, S3 removes the restored object, leaving only the S3 Glacier copy. For a transcoding job, you only have to restore objects long enough to complete the transcode job, at which point you or S3 can remove the restored object.<\/p>

Select the retrieval time required, either\u00a0Bulk or Standard<\/strong>.Choose\u00a0Next<\/strong>.<\/p>

In step 3, use the same options described before, except the IAM role. In the case of an S3 Glacier restore job, the IAM role also needs\u00a0s3:RestoreObject<\/em>permissions for the bucket containing the S3 Glacier object.<\/p>

Complete the job creation and confirmation steps as before.<\/p>

This new function activates when a client submits an S3 Glacier restore job. Based on the retrieval time selected, S3 restores the objects from S3 Glacier. Each completed object restore function then triggers the VOD automation workflow and completes the transcode jobs in MediaConvert.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Transcoding video files with S3 Batch Operations Many media companies store their extensive video repositories in S3. Typical practice suggests storing current videos in S3 Standard and archiving older videos into S3 Glacier. Lambda invocations could be triggered on S3 events such as putting a new object into a bucket. S3 Batch Operations provides you […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[25,1],"tags":[],"class_list":["post-4897","post","type-post","status-publish","format-standard","hentry","category-blogs","category-uncategorized"],"acf":[],"_links":{"self":[{"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/posts\/4897","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/comments?post=4897"}],"version-history":[{"count":3,"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/posts\/4897\/revisions"}],"predecessor-version":[{"id":14849,"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/posts\/4897\/revisions\/14849"}],"wp:attachment":[{"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/media?parent=4897"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/categories?post=4897"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/petabytz.com\/blogs\/wp-json\/wp\/v2\/tags?post=4897"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}