"CUDA driver version is insufficient" on RunPod: how to fix it

“CUDA driver version is insufficient for CUDA runtime version” means the pod’s software was built for a newer CUDA than the host machine’s NVIDIA driver supports. On RunPod you can’t update that driver, so you deploy again with the CUDA Versions filter set: 13.0 and newer for our LoRA trainer, whose image uses CUDA 13 wheels. It’s one of the failure cases in our RunPod for AI images guide.

What does “CUDA driver version is insufficient” mean?

Two pieces of software have to agree. The NVIDIA driver lives on the host machine RunPod rents you. The CUDA runtime ships inside the template’s container image, with PyTorch and the app. When the image is newer than the driver, CUDA refuses to start. NVIDIA calls this an unsupported configuration: the driver is older than the runtime library.

NVIDIA’s runtime API docs describe the error code exactly that way, and the fix they give is a newer driver. On your own PC you’d install one. On a pod the driver isn’t yours to change, so you change machines instead.

The other direction is fine. NVIDIA’s compatibility docs say a newer driver runs software built for an older CUDA. That’s why the fix is always “newer host”, never “older host”.

Which messages mean the same problem?

You’ll rarely see the exact NVIDIA wording. PyTorch, Docker and RunPod each report the same mismatch in their own words, at different moments: some before the container even starts, some only when the app first touches the GPU. All of the messages in the table below have the same fix: redeploy the pod on a machine with a newer driver.

The PyTorch number is NVIDIA’s version format: 1000 × major + 10 × minor, so 12080 is 12.8 and 13000 is 13.0 (NVIDIA, PyTorch source, checked 2 Oct 2026).

How do you fix it on RunPod?

Deploy a fresh pod on a machine whose driver supports the CUDA version you need. RunPod’s docs place the filter under Additional filters → CUDA Versions; if your deploy page looks different, open its Filter button and look for CUDA. Set it, then pick your GPU from what’s left.

1. Read the log line. Note the error and, if PyTorch printed one, the “found version” number.
2. Rescue anything you need. If the pod ran before and holds files, download them first. A pod that never started has nothing on it.
3. Terminate the broken pod. Stopping it keeps billing its disk, and restarting puts it back on the same old machine.
4. Deploy the template again. Before choosing a GPU, open the filters and set CUDA Versions.
5. Tick the version and everything above it. For our LoRA trainer: 13.0 and every newer version listed. RunPod gives this advice for Serverless endpoints (RunPod docs, checked 2 Oct 2026), and the same logic holds for pods: CUDA is backward compatible, so a wider range gives you more machines.
6. Pick the GPU and deploy. If your first card is gone, take the next one in the same VRAM tier.
7. Watch the log. The error should be gone. First boot still has to download its models.

The CUDA filter is in RunPod’s manage pods docs, with the exact advice: if you see “OCI runtime create failed”, use the CUDA Versions filter to select compatible machines.

Which CUDA version does your template need?

Check the template’s README first. Ours says it. The LoRA trainer’s README states the image uses CUDA 13 wheels and that if the log says the driver is too old, you redeploy with the CUDA filter set to 13.0. If the README is silent, the image tag often names the CUDA version.

What each CUDA generation needs from the host, from NVIDIA’s release notes (checked 2 Oct 2026):

These are the minimums for running at all, under NVIDIA’s minor-version compatibility. Some newer CUDA 13 features need a newer driver: NVIDIA lists CUDA 13.4’s new features as needing an R615 driver or later. That’s another reason to tick every newer version.

If a pod does start, you can see what the host offers. Open the web terminal and run nvidia-smi. The top line shows the driver version and the highest CUDA version it supports, like NVIDIA’s example Driver Version: 580.65.06 CUDA Version: 13.0 (NVIDIA).

Why not downgrade PyTorch inside the pod instead?

Because on a template it doesn’t stick. Anything you install outside /workspace lands on the container disk, which RunPod erases when the pod stops. You’d redo it on every start, and you’d be running a different PyTorch than the template’s author tested. Redeploying with the filter changes nothing inside the template.

Our trainer is a good example. Its image is rebuilt on every push and every Monday on the newest official AI Toolkit release. Pinning an old PyTorch inside it would fight that. The CUDA filter is the fix the trainer’s README gives.

If you write your own scripts on a bare PyTorch template, matching PyTorch to the host is a fair option. That’s a different job from running a ready-made template.

What can go wrong?

• You restart instead of redeploying. A stopped pod is tied to its machine, so it comes back on the same old driver with the same error.
• The filter leaves no card in your tier. Take the next card up in VRAM, or wait. Don’t drop to a smaller card: you’ll trade a CUDA error for an out of memory error.
• The log shows a different error after the fix. First boot is still downloading, or it’s a separate problem. Work through the stuck pod checklist.
• You leave the broken pod stopped. Its volume disk keeps billing while stopped. Terminate it once you’ve taken anything you need.