Google Cloud Setup
Prerequisites
- A GCP (Google Cloud Platform) account and access to the admin console.
- An existing project in your GCP account that you will use with Hybrik.
GCP Network Setup
Hybrik will launch render node machines via GCE (Google Compute Engine) directly in your GCP account. These instances are launched into a VPC Network configured in your GCP account that defines the virtual network structure. The Hybrik GCP Computing group supports specifying which VPC Network and which region we’ll use to launch machines in. Your VPC Network must contain subnets for the desired region. Hybrik will detect these subnets automatically and use them to launch machines.
For PoC testing, we suggest using the ‘default’ VPC network, which is the default network provisioned in GCP that has subnets in all regions. If using this network, you can skip to GCP Service Account Creation.
The VPC Network can be set in the Google CLoud Console under: Networking > VPC Network > VPC Networks
If you choose to create a new VPC network, the following settings must be applied:
- You must have at least one subnet configured for the region you intend to utilize
- The subnet(s) must support an IP range >= max instances to be launched. An example IP Range is “10.10.0.0/16” which allows for 10.10.0.0 - 10.10.255.255.
- For support purposes during onboarding, it may be necessary for Hybrik Support to SSH into Hybrik launched GCE instances in your account. To allow this, the following firewall rule must be added to the VPC network.
- Navigate to Google Cloud Console: Networking > VPC Network > Firewall Rules.
- Click Create Firewall Rule
- In the resulting form, fill out the details for a new ingress rule as defined below:
- Name: hybrik-allow-support-ssh
- Logs: Off
- Network: choose the custom network you created
- Priority: 1000
- Direction of Traffic: Ingress
- Action on match: Allow
- Targets: All instances in the network
- Source Filter: IP Filter
- Source IP Ranges: 54.160.171.213/32
- Second Source Filter: None
- Protocols and ports -> Specified protocols and ports: tcp: 22
- Click ‘Create’
GCP Service Account Creation
In order for Hybrik to manage compute resources and access storage in your GCP account, you’ll need to create a Service Account.
- Navigate to Google Cloud Console: IAM > IAM & Admin > Service accounts
- Click "+ Create Service Account" at top of screen.
- Enter a service account name, e.g. “hybrik-cg” and click create.
- You’ll be asked which roles to assign the the service account. Add the following roles:
- Compute Admin
- Storage Admin
- Storage Object Admin
- You’ll be taken to the “Grant users access to this service account” step. Click on the ‘Create Key’ button, choose the JSON key type when prompted, click Create. This will download a JSON file containing the service account credentials. These credentials will later be input into Hybrik to enable Hybrik to launch machines in your account.
- Click the Done button
- You’ll be taken back to the Service Account list which will have the new account listed. Copy the service account email address, e.g. “my-service-user@myproject.iam.gserviceaccount.com” and provide it to your Hybrik support contact. The service account user must be whitelisted on our side to be able to utilize Hybrik + GCP.
GCP Quotas
GCP has quotas which limit the max number of machines, CPUs, etc that you can run in your account. By default, these limits are very low and will interfere with reasonable operation. You must request increases on a few quota limits:
- Navigate to Google Cloud Console: IAM > IAM & Admin > Quotas
- Filter the quota list ‘Locations’ to the region you’ll be running in.
- Filter the Metrics to: CPUs, In-use IP addresses, and Persistent Disk SSD (GB)
- You should see a list of these 3 quotas. Checkmark all of them, and click Edit Quotas. Request the following quotas:
- CPUs: 10000
- In-use IP Addresses: 250
- Persistent Disk SSD (GB): 75000 (These quotas are for PoC testing purposes. Your actual quota needs for production may differ)
Until the quota increase requests from step 3 above are granted, you will be severely limited in capabilities. It’s best to wait until the quota limits are granted before proceeding to step 4.
Hybrik Credentials Vault
The Credentials Vault is used to securely store encrypted credentials that will be used by the Hybrik system to manage machines and data. These credentials may be for different GCP accounts (for example, you might use one account for machine creation and another account for storage), or even for different cloud providers.
- Login to the Hybik web interface and navigate to Account > Credentials Vault.
- Click New Credentials
- Give the credentials a name, eg “Google GCP Test”
- Give the credentials an API Key, eg “google-test”. This Api Key is used in the context of Hybrik jobs to reference this set of credentials when defining sources, destination locations, etc. Using an API Key allows you to reference various credentials in jobs without including them in cleartext.
- Set Type to: Google Cloud Platform
- In the Service JSON field, paste the contents of the Service Account JSON for your GCP Service Account.
- Click Save
- Hybrik Computing Group
Now that you have a set of GCP credentials in your Credentials Vault, you can proceed with configuring a Computing Group. A Computing Group defines the location and type of machine that Hybrik will be managing. Multiple Computing Groups can be created for more sophisticated management.
- In the Hybrik UI, navigate to Machines > Configuration.
- Click New Computing Group > GCP
- Name: Enter a name for the computing group, eg “GCP Test”
- Credentials: Select your GCP credentials vault entry in the Credentials dropdown
- VPC Network Name: Enter the name of the VPC Network from your GCP account that should be used. Type in “default” to use the GCP default network, or type in the name of your custom network.
- GCE Region: Choose the region you wish to operate in. This should be the same region your GCP storage is in.
- Group Type: On Demand
- Instance Type: n1-standard-4
- Max instances: 10
- Click Save
Now you can begin running jobs! In the Hybrik UI, go to the Jobs - Active page and select the "Submit Job JSON" button. Select a JSON file for submission. A sample JSON is shown in the next section.
Sample Job JSON
Below is some sample JSON showing a job to be run on Hybrik. The job specifies a "credentials_key" for both the source and destination locations. This is the key that was set up in the Credentials Vault.
{
"definitions": {
"profile_name": "sample_gcp_job",
"source": "gs://my_bucket/my_input_folder/my_file.mp4",
"destination": "gs://my_bucket/my_output_folder"
},
"name": "{{profile_name}}: {{source}}",
"payload": {
"elements": [
{
"uid": "source_file",
"kind": "source",
"payload": {
"kind": "asset_url",
"payload": {
"storage_provider": "gs",
"url": "{{source}}",
"access": {
"credentials_key": "google_test"
}
}
}
},
{
"uid": "transcode_task",
"kind": "transcode",
"payload": {
"location": {
"storage_provider": "gs",
"path": "{{destination}}",
"access": {
"credentials_key": "google_test"
}
},
"targets": [
{
"file_pattern": "{{profile_name}}{default_extension}",
"existing_files": "replace",
"container": {
"kind": "mp4"
},
"video": {
"codec": "h264",
"width": 512,
"height": 288,
"bitrate_mode": "vbr",
"bitrate_kb": 800,
"max_bitrate_kb": 960
},
"audio": [
{
"codec": "aac_lc",
"channels": 2,
"sample_rate": 48000,
"bitrate_kb": 96
}
]
}
]
}
}
],
"connections": [
{
"from": [
{
"element": "source_file"
}
],
"to": {
"success": [
{
"element": "transcode_task"
}
]
}
}
]
}
}