GPU cost optimization is the practice of lowering what you pay for each unit of useful GPU work. GPU monitoring collects telemetry, the measurements each GPU reports about itself over time, and connects it to cost and owners.
Most bills count in GPU-hours. A GPU-hour is one GPU running for one hour, whether it computes or waits. Utilization is the share of that time the GPU spends doing work.
Skip the monitoring half and you end up optimizing the rate on GPUs that sit idle. A cheaper hour on a GPU nobody uses is still an hour of waste.
The question underneath is rarely "what did we spend on GPUs?" It's closer to this: are we putting GPU capacity behind the products and teams that earn it? GPU cost monitoring is how you answer that with numbers instead of opinions.
GPU efficiency also sets the production cost of AI. It decides what each token, training run, or inference costs to make, and that cost flows into feature cost and margin.
GPU-hours are one part of AI cost, not all of it. Storage, data transfer, orchestration, evaluations, and the people running the work sit outside the GPU line.
GPU spend breaks a few assumptions that work fine for CPU instances. Three differences matter most for cost:
The first point comes straight from how Kubernetes schedules hardware. Device plugins expose GPUs to Kubernetes as extended resources, and the Kubernetes docs describe the rules for extended resources in general.
The Kubernetes extended resource rules state: "Extended resources cannot be overcommitted, so request and limit must be equal if both are present in a container spec." The same page says the API server "restricts quantities of extended resources to whole numbers."
In practice, a job that needs a quarter of a GPU pays for all of it.
If you want the longer argument, read about where traditional FinOps breaks down once GPU costs show up.
The goal of GPU monitoring is to tell a busy GPU from a useful one.
You also need to see waste per GPU, not per instance. One idle card hides easily on a big node.
The utilization number you see in nvidia-smi comes from NVML, NVIDIA's management library. NVIDIA's NVML utilization definition calls it "Percent of time over the past sample period during which one or more kernels was executing on the GPU."
A kernel is a function the GPU runs. The definition measures time busy, not work done.
One small kernel running nonstop reads as 100%, even if most of the chip sits unused. A GPU at 100% can still waste money. A GPU at 0% for days is the easy catch.
Treat utilization as a first filter, then confirm it with deeper signals.
An SM, or streaming multiprocessor, is one of the compute units that make up a GPU chip.
The table compares five GPU signals by what each one tells you and the cost decision it supports.
| Signal | What It Tells You | Cost Decision It Supports |
|---|---|---|
| GPU utilization | Whether any kernel ran during the sample period | Catch GPUs that sit fully idle |
| SM activity | How much of the chip is actually working | Find busy but underused GPUs to share or downsize |
| GPU memory used and memory utilization | Whether the model fits, and with how much room | Move to a smaller GPU or a slice of one |
| Power draw | Whether a powered-on GPU is doing real work | Confirm idle before you shut anything down |
| CPU utilization next to GPU | Whether the bottleneck sits outside the GPU | Check whether the job needs a GPU at all |
Keep every signal per GPU, not averaged per instance. Take an eight-GPU node where seven run flat out and one does nothing: the instance average reads about 88%, and the idle card disappears.
Pick the collection path that matches where your GPUs run. Then make sure the data carries the labels you need for ownership: namespace and pod in Kubernetes, instance ID on EC2.
Telemetry has a cost of its own. Budget for metric storage and ingestion before you collect every signal at the shortest interval.
DCGM is NVIDIA's Data Center GPU Manager. DCGM Exporter is the companion tool that publishes GPU telemetry as Prometheus metrics, the open-source monitoring format many Kubernetes teams already collect.
You want the exporter running on every GPU node you pay for. If one node pool is missing it, that pool's GPUs drop out of every report built on top.
On GKE, Google handles most of the install. Google's GKE documentation describes GKE's managed DCGM metrics this way: "When you enable DCGM metrics, GKE installs the DCGM-Exporter tool, installs Google-managed GPU drivers, and deploys a ClusterPodMonitoring resource to send metrics to Google Cloud Managed Service for Prometheus."
The same Google documentation defines SM activity (DCGM_FI_PROF_SM_ACTIVE) as "The ratio of cycles an SM has at least 1 warp assigned." A warp is a group of threads the SM runs together, and SM activity is the profiling metric to add beyond utilization.
Pod and namespace labels are the bridge to allocation. Without them, you know a GPU was busy but not who kept it busy.
Most setup problems show up later as missing nodes or dropped labels:
Finout is an enterprise-grade FinOps platform for cloud and AI spend. Finout's EC2 GPU metrics setup guide notes that CloudWatch doesn't emit GPU usage metrics by default, and that the CloudWatch Agent must be installed and configured on each GPU instance.
AWS Systems Manager can deploy the agent at scale. The same guide points out that AWS embeds GPU cost into EC2 instance pricing with no dedicated cost dimension.
That's why usage metrics are the only way to see GPU waste on EC2. The bill shows the instance, and only telemetry shows what the GPUs inside it did.
Set InstanceId as a metric dimension. That dimension ties each GPU metric to the EC2 resource you're billed for.
Keep metrics per physical GPU, identified by instance ID plus GPU index. The guide calls this "critical for detecting partial waste on multi-GPU instances."
Budget for the telemetry, too. GPU metric collection through the CloudWatch Agent incurs standard CloudWatch custom metric charges. Finout's GPU scans also need at least 7 days of GPU metrics before they return their first results.
Effective GPU cost is your hourly cost divided by the share of time the GPU does useful work. It explains why the bill feels higher than the rate.
As a hypothetical, say an instance costs $40 an hour and does useful work 25% of the time. Each useful hour costs $160.
Price per GPU-hour is an input metric. It describes what you paid for hardware, not what the hardware delivered.
Output metrics describe what you got. Pick the one that matches the workload:
A cheaper GPU that runs half as fast isn't cheaper. If the same run takes twice the hours, cost per training run can rise even as the hourly rate falls.
Your inference software matters as much as the hardware. The serving stack decides how much of a GPU's capacity becomes delivered tokens, which means today's cost per token isn't a fixed floor.
This is where GPU efficiency reaches the business. It sets production cost, which sets cost per feature, which can reach the price you charge customers.
Idle GPUs are the cheapest waste to fix. Removing them doesn't touch the performance of anything that's running.
Finout Product Manager Lior Blumenfeld describes the two faces of Kubernetes waste and the pattern behind them. Across the organizations Finout works with, on average, over 70% of Kubernetes spend turns out to be waste once it's measured properly.
That figure covers Kubernetes broadly, not GPUs specifically. Applied to GPUs, Finout's interpretation is that GPU waste also comes in two kinds with two owners.
Node-level idle is a GPU node with no work scheduled on it. Platform teams own it, because they decide how many nodes exist.
Workload over-requesting is a pod holding whole GPUs it barely uses. Developers own that one, because they write the resource requests.
Blend the two into one "idle" number and nobody owns it. Split them, and each owner gets a number they can act on.
A concrete idle test helps. Finout's GPU Idle scan, described in the EC2 GPU metrics setup guide above, looks back 7 days and flags an instance only when every GPU meets every threshold on every day.
Each GPU must average under 10% utilization, under 20% memory utilization, and under 15% power draw. CPU must peak below 5%, and average network traffic in and out must stay under 30 MB.
The multi-day window matters. A training job between epochs (full passes over the training data) can look idle for an hour. Seven straight days of nothing is a different story.
The same scans flag a second pattern. When GPU utilization sits under 10% while CPU peaks above 50%, the recommendation is an equivalent non-GPU instance.
Finout's documentation puts it bluntly: high CPU with low GPU means a misconfigured workload. The savings equal the cost difference between the GPU instance and its CPU replacement.
Most idle GPU time traces back to four patterns, and each one needs a different fix:
Once idle capacity is gone, the next question is whether each job runs on the right amount of GPU.
That means picking the right GPU first, then sharing it when one job can't fill it.
Training and inference want different things. Training cares about throughput and the interconnect between GPUs, since big jobs split work across many cards. Most inference cares about memory fit and latency.
Profile before you choose. Run a representative job, record its memory footprint and SM activity, then pick the smallest GPU that meets your target.
Judge the options on cost per training run or cost per 1,000 inferences, not price per hour. A larger GPU that finishes in a third of the time can win on output cost.
Revisit the choice after a model or framework change. A new version can shift memory use enough to fit a smaller GPU, or push you past the one you have.
Multi-Instance GPU, or MIG, splits one physical GPU into isolated hardware slices. NVIDIA's MIG user guide says NVIDIA Ampere and later GPUs can be "securely partitioned into up to seven separate GPU Instances for CUDA applications."
Time-slicing is software sharing, where several workloads take turns on the same GPU. It's flexible, and it gives up most of the guardrails MIG provides.
NVIDIA's time-slicing documentation warns that "there is no memory or fault-isolation between replicas." It also states that "DCGM-Exporter does not support associating metrics to containers when GPU time-slicing is enabled with the NVIDIA Kubernetes Device Plugin."
The table compares whole GPUs, MIG, and time-slicing by how they share, how they isolate, how cost gets attributed, and where each fits.
| Option | How It Shares | Isolation | Cost Attribution | Best Fit |
|---|---|---|---|---|
| Whole GPU | No sharing | Full isolation | Simple: one workload, one owner | Large training jobs |
| MIG | Fixed hardware slices | Memory and fault isolation per slice | Slice-level attribution | Small inference and dev work |
| Time-slicing | Workloads take turns | No memory or fault isolation | No per-container GPU metrics from DCGM-Exporter | Bursty, low-priority jobs |
If you time-slice, decide how you'll allocate cost before you turn it on. Once per-container GPU metrics disappear, you'll need a rule, such as allocating by node or by replica configuration.
If utilization is still low, fix it before you touch the rate.
A lower rate on idle GPUs only makes waste cheaper, and a commitment locks that waste in for the full term.
Spot capacity, called preemptible on some clouds, is spare capacity sold at a discount that the provider can reclaim. It fits work that can stop and restart without anyone noticing.
Good fits are fault-tolerant training with checkpoints, batch inference, and experiments. A checkpoint is a saved snapshot of a job's state that lets it resume where it stopped. Latency-sensitive production inference is a poor fit.
Plan for a short warning. Azure's Scheduled Events notice gives a Spot VM preemption a 30-second minimum notice, which "is made available on a best effort basis." Check your own provider's notice window before you design around it.
Don't build recovery around the warning. Checkpoint on a schedule that doesn't depend on it, keep checkpoints in durable storage, and make every job resumable.
Add retry logic that relaunches across zones or regions.
Commit only the steady baseline you can see in at least a few weeks of utilization data. Production inference that runs around the clock is the usual candidate.
Keep bursty training and experiments on on-demand or spot. Watch model churn too: a commitment sized for today's GPU type can outlive the workload it was bought for.
On Azure, a billing commitment isn't a capacity guarantee. Microsoft's documentation on Azure Reserved VM Instances is explicit: "Reserved VM Instances provide a billing discount only and do not reserve or guarantee compute capacity."
That distinction matters when GPUs are scarce. Check whether your provider's reservations hold capacity or only lower the rate, and treat guaranteed GPU capacity as a separate decision.
For finance partners, commitment coverage should follow measured utilization, not forecasts alone.
A shared GPU cluster produces one bill and many users. Allocation maps that bill to teams, models, and features, and that mapping makes showback, budgets, and the investment conversation possible.
Showback means reporting costs back to the teams that drove them, without billing them internally. It only works when every GPU has an owner.
Most of these steps are policy decisions, and the first one decides whether the rest hold:
For the full method, see the guide to allocating GPUs in shared clusters.
Finout's page on Kubernetes cost allocation by team covers the agentless connection to Prometheus or AWS CloudWatch metrics. From there, you can drill from cluster to namespace, deployment, label, and pod.
Tenable's pod-level cost insight shows what that drill-down looks like in practice. Vijay Kurra, Senior Manager Cloud DevFinOps at Tenable, described the result:
"Finout's advanced visibility capabilities go beyond basic cluster billing, allowing us to identify and address cost inefficiencies with precision and accountability."
Most of these mistakes come from acting on one signal, or on a signal nobody owns:
The right first move depends on where your GPUs run and who's asking:
The next question to ask is short. Which of our GPU workloads has an owner, a unit metric, and a target?
Finout is an enterprise-grade FinOps platform for cloud and AI spend, built for the agentic era. Finout puts GPU telemetry next to cost and owner, which turns a utilization chart into a decision someone can make.
Billy's GPU utilization prompt asks which GPU instances are underutilized this month. The answer ranks instances with GPU utilization, GPU memory utilization, and power draw next to cost. That view shows which instances are genuinely idle rather than just quiet.
Instances already flagged by the GPU Idle scan in CostGuard are called out separately.
Finout's AI capabilities split by who asks the question and who acts on the answer:
The cost and allocation layer handles ownership and follow-through:
If you want to see GPU utilization, cost, and owner side by side, Book a demo.