Jobs and Tasks

Jobs and Tasks

A Job is defined in JSON notation, and tells the Hybrik service precisely how to execute that Job. Note that a Job is made up of one or more Tasks, and Tasks are then distributed across the available machines. Therefore a Job may end up executing on many different machines. Both Jobs and Tasks can have varying priorities within the system to allow you to finely control performance and latency. Tasks may even be set to retry if they fail, which allows you to manage intermittent error conditions like network accessibility. Depending on your workflow, some Tasks may not get executed. For example, a Job execution might look like this:

• Task 1 - Analyze a source file
• Task 2 - Analyze the source file
• Task 3 - Verify that the audio levels match the input requirements
  • if pass, go to Task 4
  • if fail, go to Task 7
• Task 4 - Transcode source file to MP4
• Task 5 - Copy result to CDN
• Task 6 - Notify user that result is live and stop here
• Task 7 - Copy failed source file from Task 2 to new location
• Task 8 - Notify QC department of failure

If the file passes the QC test, Tasks 1, 2, 3, 4, 5, and 6 execute. If the file fails the QC test, Tasks 1, 2, 3, 7, and 8 execute. A Job specifies not only which Tasks will be executed, but also how those Tasks are connected to each other.

Job Structure

The basic structure of a Job JSON looks like this:

• Job Info
  • Name
  • Priority
  • Etc.
• Task Element Info
  • Task 1
  • Task 2
  • Etc.
• Connection Info
  • Task 1 is connected to Task 2
  • Task 2 is connected to Task 3 on success, but Task 4 on failure
  • Etc.

A sample JSON file describing a Job is shown below. This Job takes a source file from an Amazon S3 location (s3://hybrik-examples/public/sources/sample1.mp4) and puts the transcode result into a different S3 location (s3://hybrik-examples/public/output/example1). The transcode video parameters are set to use the h.264 codec, with a width of 640 pixels, a height of 360 pixels, a frame rate of 23.976 frames per second, and a bitrate of 600kb/sec. The audio format is set to use the HE-AAC V2 codec, with 2 channels of audio, at a sample rate of 44.1kHz and a bitrate of 128kb/sec.

Jobs may be simple and define only a Source and Transcode element, or they may be very complex with dozens of different tasks and conditional branching.

Hybrik Job

{
  "name": "Hybrik API Example #1 - simple transcode",
  "payload": {
    "elements": [
      {
        "uid": "source_file",
        "kind": "source",
        "payload": {
          "kind": "asset_url",
          "payload": {
            "storage_provider": "s3",
            "url": "s3://hybrik-examples/public/sources/sample1.mp4"
          }
        }
      },
      {
        "uid": "transcode_task",
        "kind": "transcode",
        "task": {
          "retry_method": "fail"
        },
        "payload": {
          "location": {
            "storage_provider": "s3",
            "path": "s3://hybrik-examples/public/output/example1"
          },
          "targets": [
            {
              "file_pattern": "%s.mp4",
              "existing_files": "replace",
              "container": {
                "kind": "mp4"
              },
              "video": {
                "codec": "h264",
                "width": 640,
                "height": 360,
                "frame_rate": 23.976,
                "bitrate_kb": 600
              },
              "audio": [
                {
                  "codec": "heaac_v2",
                  "channels": 2,
                  "sample_rate": 44100,
                  "bitrate_kb": 128
                }
              ]
            }
          ]
        }
      }
    ],
    "connections": [
      {
        "from": [
          {
            "element": "source_file"
          }
        ],
        "to": {
          "success": [
            {
              "element": "transcode_task"
            }
          ]
        }
      }
    ]
  }
}