Google Cloud
Connect Google Cloud with one Cloud Shell command, request GPU quota, choose storage, and verify training and recovery.
Connect Google Cloud to train on 1, 2, 4 or 8 NVIDIA GPUs of one Compute Engine VM in your project. Setup takes GPU quota, one command in Cloud Shell, and one paste into Tensorant. Tensorant then creates a VM for each run in the zone you choose, moves data through your storage, and deletes the VM after verifying the adapter.
Your own GPU accounts come with the Own compute subscription. A Google Cloud account connected before Own compute existed keeps working without it until 31 Dec 2026: see Your own accounts. Google Cloud bills the GPU time in your account, and each training run's training-token fee comes from your Tensorant credits.
Google Cloud trains only. Test sessions and published models run on Tensorant compute or RunPod.
1. Request GPU quota
Use a project with billing enabled on a paid account. Free trial accounts can't create GPU VMs.
Tensorant offers these machine types, each with its GPUs built in:
| Machine type | GPUs | vCPUs | About, in us-central1 |
|---|---|---|---|
g2-standard-8 |
1 × NVIDIA L4, 24 GB | 8 | $0.85/h |
g2-standard-24 |
2 × NVIDIA L4, 24 GB each | 24 | $2.00/h |
g2-standard-48 |
4 × NVIDIA L4, 24 GB each | 48 | $4.00/h |
g2-standard-96 |
8 × NVIDIA L4, 24 GB each | 96 | $8.00/h |
a2-highgpu-1g |
1 × NVIDIA A100, 40 GB | 12 | $3.67/h |
a2-highgpu-2g |
2 × NVIDIA A100, 40 GB each | 24 | $7.35/h |
a2-highgpu-4g |
4 × NVIDIA A100, 40 GB each | 48 | $14.69/h |
a2-highgpu-8g |
8 × NVIDIA A100, 40 GB each | 96 | $29.39/h |
a2-ultragpu-1g |
1 × NVIDIA A100, 80 GB | 12 | $5.07/h |
a2-ultragpu-2g |
2 × NVIDIA A100, 80 GB each | 24 | $10.14/h |
a2-ultragpu-4g |
4 × NVIDIA A100, 80 GB each | 48 | $20.28/h |
a2-ultragpu-8g |
8 × NVIDIA A100, 80 GB each | 96 | $40.55/h |
a3-highgpu-8g |
8 × NVIDIA H100, 80 GB each | 208 | $88.49/h |
Training GPU shows the machine types your zone offers with the run's Number of GPUs, priced for the whole VM. These are approximate us-central1 prices: other regions can cost 5 to 20 percent more. Google sells H100 VMs of 1, 2 or 4 GPUs only as Spot or Flex-start VMs, which Tensorant doesn't use.
Choose a zone that offers the machine type you want, such as us-central1-a. GPU regions and zones
In IAM & Admin → Quotas & System Limits, request:
- GPUs (all regions), at least the number of GPUs of the VM;
- the GPU quota of the zone's region for your GPU, such as NVIDIA L4, NVIDIA A100, NVIDIA A100 80GB or NVIDIA H100 GPUs (search the quotas for the GPU's name);
- enough vCPUs in that region for the machine type.
New projects often start with a GPU quota of 0. A granted quota does not reserve GPU capacity. GPU quota
2. Run the setup command
Use a Google Cloud project for Tensorant only. The setup grants Tensorant Compute Instance Admin (v1) on the whole project, a role that reaches every VM, disk, snapshot and image in it, including ones Tensorant didn't create. Tensorant only manages its own VMs, labelled tensorant-managed=training, but a project of its own keeps the role away from your other machines and data. Create one with IAM & Admin → Create a project, and enable billing for it.
In Tensorant, open your name at the foot of the sidebar, then Settings → Connections under Organization, and choose Connect Google Cloud. Enter the Zone. The dialog shows the command for that zone.
Open Cloud Shell with the project you will train in selected. If another project is selected, switch first with gcloud config set project PROJECT_ID. Run the command. For us-central1-a it reads:
ZONE=us-central1-a
PROJECT=$(gcloud config get-value project 2> /dev/null)
ACCOUNT=tensorant-training@$PROJECT.iam.gserviceaccount.com
[ -n "$PROJECT" ] || { echo "Select a project: gcloud config set project PROJECT_ID"; false; } &&
gcloud services enable compute.googleapis.com --project "$PROJECT" &&
{ gcloud compute firewall-rules describe tensorant-training-deny-ingress --project "$PROJECT" > /dev/null 2>&1 ||
gcloud compute firewall-rules create tensorant-training-deny-ingress --project "$PROJECT" --network default \
--direction INGRESS --action DENY --rules all --source-ranges 0.0.0.0/0 --target-tags tensorant-training --priority 100; } &&
{ gcloud iam service-accounts describe "$ACCOUNT" --project "$PROJECT" > /dev/null 2>&1 ||
gcloud iam service-accounts create tensorant-training --project "$PROJECT" --display-name "Tensorant training"; } &&
gcloud projects add-iam-policy-binding "$PROJECT" --member "serviceAccount:$ACCOUNT" \
--role roles/compute.instanceAdmin.v1 --condition=None --quiet > /dev/null &&
gcloud iam service-accounts keys create tensorant-key.json --iam-account "$ACCOUNT" --project "$PROJECT" &&
jq -c --arg zone "$ZONE" '{provider: "gcp", zone: $zone, service_account_key: tojson}' tensorant-key.json &&
rm tensorant-key.jsonIt:
- stops with "Select a project" when Cloud Shell has none selected;
- enables the Compute Engine API in the project;
- creates the firewall rule
tensorant-training-deny-ingress, unless it exists: on thedefaultnetwork, it denies all inbound traffic from anywhere to VMs with the network tagtensorant-training, which every training VM has, at priority 100; - creates the service account
tensorant-training, unless it exists; - grants it Compute Instance Admin (v1) on the project;
- creates a key of it and prints one JSON object with the zone and the key, then deletes the key file from Cloud Shell.
Run it as someone who can enable APIs, create service accounts and their keys, and change the project's IAM policy, such as a project Owner.
The training VMs run with no service account, so the account needs no other role. Compute Instance Admin (v1) also lets it read firewall rules, which Tensorant does to check tensorant-training-deny-ingress; it never changes them.
Running the command again is safe: what exists is kept. If add-iam-policy-binding fails right after a new service account was created, because Google hasn't finished creating it yet, wait a minute and run the command again.
If keys create answers that key creation is not allowed, your organization enforces the policy iam.disableServiceAccountKeyCreation, which Google enables by default in new organizations. Ask an organization policy administrator to allow keys for this project. Service account key policies
A key doesn't expire unless your organization sets a lifetime, and Google allows 10 keys a service account. Each run of the command creates another key of the same account.
3. Connect in Tensorant
Back in the Connect Google Cloud dialog, paste everything the command printed and choose Connect Google Cloud. You can also paste a key file you made yourself; then choose the zone in the dialog. Tensorant checks the key with Google, that the Compute Engine API is enabled and the zone exists, finds the project's default network, checks that the firewall rule tensorant-training-deny-ingress exists and denies as the command makes it, finds Google's GPU images, and lists the zone's GPU machine types with their prices. It does not start a VM or check your quota. It checks the firewall rule again before each launch, and refuses to launch without it.
Allow a minute after running the command: a new role can take that long to apply. If the check fails right away, wait and try again.
To change the zone or the key later, choose Update connection in the provider's ... menu. Tensorant refuses a new zone or key while a run, or a VM kept for recovery, is in the old one: finish or clean them up first.
What Tensorant sets up
Each training VM gets:
| Part | Setting |
|---|---|
| Image | A2 and A3: Google's Deep Learning VM image family common-cu129-ubuntu-2404-nvidia-580. G2, which Deep Learning VM images don't boot: Google's Ubuntu accelerator image family ubuntu-accelerator-2404-amd64-with-nvidia-580. Both are Ubuntu 24.04 with NVIDIA driver 580, their latest image. |
| Network | The project's default VPC network on Google's gVNIC network interface, with an external IP for downloads |
| Firewall | The network tag tensorant-training, which the rule tensorant-training-deny-ingress denies all inbound traffic. Its priority, 100, comes before the default network's own rules (such as default-allow-ssh, at 65534), so nothing from outside reaches the VM. Its outbound connections, and their answers, are not affected. No SSH key is set on the VM. |
| Identity | No service account and no access scopes: the VM has no Google Cloud permissions |
| Maintenance | Host maintenance stops the VM, which GPUs require, and it isn't restarted |
| Disk | A balanced persistent boot disk of the run's disk size, at least the image's size, deleted with the VM |
| Labels | tensorant-managed=training and tensorant-run=tune-RUN_ID; the VM is named tune-RUN_ID |
| Startup | A startup script that starts the training container on the first boot only. It adds Docker and the NVIDIA Container Toolkit if the image lacks them. |
The startup script carries your storage connection's keys, and anyone who can view the VM in your project can read it. Give the storage connection keys limited to its bucket.
Storage
Choose one before creating projects:
- Tensorant storage, with nothing to set up. The VM reaches the run's files through links Tensorant gives it, so a run lasts at most 6 days.
- A Cloud Storage bucket of yours, connected as S3-compatible storage. Cloud Storage accepts S3 requests signed with HMAC keys. Create the bucket in the region of your zone, a service account that may use only that bucket, and an HMAC key of it:
PROJECT=$(gcloud config get-value project 2> /dev/null)
BUCKET=my-team-tensorant-data
gcloud storage buckets create gs://$BUCKET --location us-central1 \
--uniform-bucket-level-access --public-access-prevention &&
gcloud iam service-accounts create tensorant-storage --display-name "Tensorant storage" &&
gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
--member "serviceAccount:tensorant-storage@$PROJECT.iam.gserviceaccount.com" --role roles/storage.objectAdmin &&
gcloud storage hmac create tensorant-storage@$PROJECT.iam.gserviceaccount.comThen in Connections → Data storage, choose custom S3 and enter:
| Tensorant field | Cloud Storage |
|---|---|
| Bucket | Your bucket's name |
| Signing region | auto |
| Service endpoint | https://storage.googleapis.com |
| Addressing style | path |
| Access key ID | The HMAC key's access ID |
| Secret access key | The HMAC key's secret, shown once |
Cloud Storage interoperability
4. Verify a training trial
- Choose your storage, then create a project and a small dataset version.
- In Configure a run, choose Training provider → Google Cloud, the Number of GPUs and a Training GPU with enough memory.
- Choose Training trial · a small sample and set a Price limit ($/hour). Allow 10 to 20 minutes for the first image and model downloads.
- Save the draft, review the cost confirmation and launch. Google Cloud starts charging for the VM.
- Confirm training and adapter verification succeed. Then check GPU released in Tensorant, and that the VM
tune-RUN_IDand its disk are gone from Compute Engine → VM instances.
The estimate is the machine type's approximate on-demand list price in us-central1, plus the balanced boot disk at $0.10 a GB-month divided by 730. Other regions can cost 5 to 20 percent more than these us-central1 figures, and the external IP, network traffic, taxes and your discounts are outside it. The price limit is checked against that estimate; it is not a Google Cloud billing cap. See training runs for the trial flow and launch limits.
Tensorant follows Google's answer to the launch for up to 90 seconds. If Google refuses it in that time, for quota or because it has no free machine of that type in your zone, the launch fails, says which, and nothing is billed. If Google is still making the VM after 90 seconds, Tensorant leaves it to Google and keeps watching the VM; should Google then fail to make it, the run fails because its machine is gone, without the reason.
Recovery, cleanup and key rotation
On a timeout or a failed export, Tensorant stops the VM and keeps its boot disk. A stopped VM shows as Stopped in the console (TERMINATED in Google's API); its GPU and vCPU charges stop, and the disk continues to bill. A2 Ultra and A3 VMs come with local SSDs, which a stop empties; Tensorant keeps nothing on them.
Open Recover your adapter on the run. To retrieve the completed archive yourself, you need a way into the VM. The rule tensorant-training-deny-ingress keeps every connection from outside away from it, SSH included. Allow SSH through Google's Identity-Aware Proxy (IAP) only, with a rule of a lower priority number for the same tag:
gcloud compute firewall-rules create tensorant-training-allow-iap-ssh --network default \
--direction INGRESS --action ALLOW --rules tcp:22 --source-ranges 35.235.240.0/20 \
--target-tags tensorant-training --priority 9035.235.240.0/20 is the range IAP connects from, and IAP lets through only people your project allows to use it, such as an Owner. Nothing else can reach the VM: every other inbound connection still meets the deny rule.
- Start the retained VM in Compute Engine → VM instances. Starting resumes billing; it does not start training again.
- Connect with
gcloud compute ssh tune-RUN_ID --zone ZONE --tunnel-through-iap. - With custom S3, download
/var/lib/tensorant/data/adapter.zip; with RunPod network storage,/var/lib/tensorant/data/runs/RUN_ID/adapter.zip. The host's data directory is/workspace/datain the container. - Delete the IAP rule with
gcloud compute firewall-rules delete tensorant-training-allow-iap-ssh, and stop the VM while you finish.
Select the archive on the run and choose Verify and recover. See failure and recovery.
After recovery succeeds, or when you cancel and discard the output, Tensorant deletes the VM with its boot disk. It refuses automatic deletion while a disk that would outlive the VM is attached, so that no data of yours is lost: detach that disk, then use Retry the GPU release. Wait for confirmed cleanup before removing the connection or changing projects.
To rotate the key, run the setup command again and update the connection with what it prints. Then delete the old key in IAM & Admin → Service accounts → tensorant-training → Keys.
Troubleshooting
| Symptom | Check |
|---|---|
| "Google Cloud did not accept this service account key" | A key created a moment ago can take a minute to work: wait and try again. Otherwise the key was deleted or disabled: run the setup command again and paste the new output. |
| "has no firewall rule tensorant-training-deny-ingress" | Run the setup command again: it creates the rule. |
| "The firewall rule tensorant-training-deny-ingress ... is not the one the setup command makes" | Someone changed the rule. Delete it with gcloud compute firewall-rules delete tensorant-training-deny-ingress, then run the setup command again. |
| The setup command says "Select a project" | Run gcloud config set project PROJECT_ID, then the command again. |
| "The Compute Engine API is not enabled" | Run gcloud services enable compute.googleapis.com in the project, wait a minute, and try again. |
| "The service account may not use Compute Engine" | The role is missing or not applied yet. Wait a minute, or grant Compute Instance Admin (v1) again. |
| "Google Cloud has no zone" | Check the zone's name, such as us-central1-a. |
| "has no VPC network named default" | Create a VPC network named default with automatic subnets, or use a project that has one. |
keys create says key creation is not allowed |
The organization policy iam.disableServiceAccountKeyCreation applies; see step 2. |
| The launch fails: "the project's GPU quota is too low" | Request the quota the message names, then launch again. |
| The launch fails: "has no ... free in ... right now" | Google has no capacity there now. Try later, or connect another zone that offers the machine type. |
| The launch fails: "An organization policy ... refuses the VM" | The constraint it names blocks the VM. constraints/compute.vmExternalIpAccess must allow external IPs for the project, and constraints/compute.trustedImageProjects must allow projects/deeplearning-platform-release and projects/ubuntu-os-accelerator-images. Ask an organization policy administrator. |
| Host validation or container startup fails | Open the VM's Serial port 1 log in Compute Engine, and check outbound access to container, model and storage hosts. |
| "Google Cloud VM disks changed" | A disk that is not deleted with the VM is attached. Detach it, then retry the GPU release. |
| GPU release fails | Check the key is still valid and the VM's activity in Logging, then use Retry the GPU release. |
Need a hand? Visit troubleshooting