Skip to content

Cloud GPUs

By Sergey Karayev and Charles Frye. Updated May 22, 2023.

Training and running neural networks often requires hardware acceleration, and the most popular hardware accelerator is the venerable graphics processing unit, or GPU.

We have assembled cloud GPU vendor pricing all into tables, sortable and filterable to your liking!

We have split the vendor offerings into two classes:

  • GPU Cloud Servers, which are long-running (but possibly pre-emptible) machines, and
  • Severless GPUs, which are machines that scale-to-zero in the absence of traffic (like an AWS Lambda or Google Cloud Function)

We welcome your help in adding more cloud GPU providers and keeping the pricing info current.

Please file an issue or make a pull request to this repo, editing this file to update the text on this page or one of the CSV files to update the data: cloud-gpus.csv for servers and serverless-gpus.csv for serverless options.

GPU Cloud Server Comparison

Notes

  • GCP does not have GPU "instances" in the same way that AWS and Azure do. Instead, any suitable machine can be connected to a configuration of GPUs. We have selected machines that are roughly equivalent to AWS options.
  • Regions were set to be the west or central parts of the United States. GPU availability depends on the region.
  • Raw data can be found in a csv on GitHub.

All prices are in $/hr.

Serverless GPUs

Notes

  • We use the classic definition of "serverless", courtesy of the original AWS announcement on serverless computing: no server management, flexible scaling, high availability, and no idle capacity. We only include services that fit this criterion in our options below.
  • Direct price comparisons are trickier for serverless offerings: cold boot time and autoscaling logic can substantially impact cost-of-traffic. Also, some providers only charge for time spent responding to requests, while others charge for other time you're using their machines, like booting or between requests (see the Idle time charged? column below).
  • Some of the providers allow configuration of CPU and RAM resources. We have selected reasonable defaults, generally comparable to the fixed offerings of other providers.
  • If you know a bit about your anticipated traffic patterns, you can use this tool to compare prices for AWS A100 GPU machines and Banana's serverless equivalent. Note that is is made by the developers of Banana, so may be biased.
  • Raw data can be found in a csv on GitHub.
  • You can find pricing pages for the providers here: Banana, Baseten, Modal, Replicate, RunPod
  • Serverless GPUs are a newer technology, so there are fewer players, the details change quickly, and you can expect bugs/growing pains. Stay frosty!

How do I choose a GPU?

This page is intended to track and make explorable the current state of pricing and hardware for cloud GPUs.

If you want advice on which machines and cards are best for your use case, we recommend Tim Dettmer's blog post on GPUs for deep learning.

The whole post is a tutorial and FAQ on GPUS for DNNs, but if you just want the resulting heuristics for decision-making, see the "GPU Recommendations" section, which is the source of the chart below.

Flowchart for quickly selecting an appropriate GPU for your needs, by Tim Dettmers

Flowchart for quickly selecting an appropriate GPU for your needs, by Tim Dettmers

GPU Raw Performance Numbers and Datasheets

Below are the raw TFLOPs of the different GPUs available from cloud providers.

Model Arch FP32 Mixed-precision FP16 Source
A100 Ampere 19.5 156 312 Datasheet
A10G Ampere 35 35 70 Datasheet
A6000 Ampere 38 ? ? Datasheet
V100 Volta 14 112 28 Datasheet
T4 Turing 8.1 65 ? Datasheet
P4 Pascal 5.5 N/A N/A Datasheet
P100 Pascal 9.3 N/A 18.7 Datasheet
K80 Kepler 8.73 N/A N/A Datasheet
A40 Ampere 37 150 150 Datasheet

GPU Performance Benchmarks

Below are some basic benchmarks for GPUs on common deep learning tasks.

Benchmark of different GPUs on a single ImageNet epoch, by AIME

Benchmark of different GPUs on a single ImageNet epoch, by AIME

Benchmark of different GPUs on a mix of tasks, by Lambda Labs

Benchmark of different GPUs on a mix of tasks, by Lambda Labs

We are excited to share this course with you for free.

We have more upcoming great content. Subscribe to stay up to date as we release it.

We take your privacy and attention very seriously and will never spam you. I am already a subscriber