Datadog Cost Optimization: One Tag Can 1000x Your Bill. Here's How to Find It and Stop the Bleed

Observability

Jul

2026

Jun

2026

If you run Datadog at scale, you have done the cleanup at least once. You audited custom metrics, trimmed log retention, added a few exclusion filters, and watched the monthly bill come down. Then a quarter later, it was back, and you could not point to a single decision that put it there.

That is the problem with Datadog cost. Your bill is not a mess you clean once. It is the output of a continuous flow of telemetry that grows every time a team ships a service, and the levers most DevOps teams use are manual interventions that never stop. You can pull every one of them correctly and still be back on the treadmill by next quarter.

This guide does both halves. It shows the technical levers that genuinely move spend, with the mechanics a platform team needs to actually act on them. Then it explains why pulling them by hand is structurally a losing game, and what controlling cost at the telemetry pipeline looks like instead.

What’s driving up the cost of your Datadog bill?

Datadog charges across many separate products, but a few drive most of the surprise line items: custom metrics, log ingestion and indexing, per-host charges, and APM span volume. The reason cost creeps is that the variable behind each of these is generated by developers shipping code, while the bill lands on a platform team that had no say in it. Nowhere is that clearer than in custom metrics.

The brutality of high cardinality on your Datadog bill

Datadog charges for custom metrics by counting unique timeseries, and a timeseries is one unique combination of metric name plus tag values.

Walk through one metric. Say a service emits request.latency, and someone tags it for what they “think” will generate better observability:

    request.latency
      service:    20 values
      endpoint:   50 values
      status:      5 values
      pod_name:  200 values  (ephemeral, autoscaled)

‍

That is one metric name. The billable count is the product of the tag cardinalities: 20 x 50 x 5 x 200 = 1,000,000 timeseries from a single metric. At Datadog’s $5 per 100 custom metrics, those million time series add up to an unpallatable Datadog bill quickly.

The killer tag is almost always the high-cardinality one, and it is almost always added without anyone realizing what it does. pod_name, container_id, user_id, request_id, trace_id as a tag: each one multiplies your bill by the number of distinct values. Drop pod_name from the example above, and you go from 1,000,000 timeseries to 5,000. Same metric, same dashboards in practice, three orders of magnitude less cost.

The takeaway for a platform team is that custom metric optimization is not “emit fewer metrics.” It is “control the cardinality of the metrics you already emit,” and the highest-leverage move is usually removing or aggregating away a single unbounded tag.

Find the offenders with the API, not the UI

Datadog exposes estimated usage and per-metric cardinality through the API, so you can script the audit and run it on a schedule.

The pattern: pull the active list of custom metrics, then query each metric’s tag cardinality and sort by what is actually billable. A minimal version against the metrics endpoints:

    # List active custom metrics over a window
    curl -s "https://api.datadoghq.com/api/v2/metrics?filter[queried]=true" \
      -H "DD-API-KEY: ${DD_API_KEY}" \
      -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"

    # For a given metric, get tag cardinality / estimated usage
    curl -s "https://api.datadoghq.com/api/v2/metrics/request.latency/estimate" \
      -H "DD-API-KEY: ${DD_API_KEY}" \
      -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"

‍Wrap that in a script that ranks every custom metric by estimated timeseries count, flag anything carrying a tag with more than a few hundred distinct values, and you have a repeatable cardinality report instead of a quarterly manual hunt. Run it in CI or on a cron and the worst offenders surface in days instead of on the invoice.

APM sampling: head vs tail, and the two bills you are conflating

APM cost confuses people because Datadog meters traces in two separate places: ingested spans (everything the agent sends up) and indexed spans (the subset retained for search via retention filters). Teams tune one and forget the other, then wonder why the bill did not move.

The bigger lever is usually sampling strategy. Default head-based sampling decides whether to keep a trace at the moment it starts, before anything interesting has happened. That means you pay to ingest a flood of routine traces and, statistically, drop the rare slow or errored ones you actually wanted. Per-service head sampling rules at least let you stop sampling a payment path the same way you sample a static asset server:

    DD_TRACE_SAMPLING_RULES='[
      {"service":"payments-api","sample_rate":1.0},
      {"service":"asset-proxy","sample_rate":0.05}
    ]'

‍Tail-based sampling flips the decision to after the trace completes, so you can keep all errors and high-latency traces and aggressively drop the fast successful ones. That is the configuration most teams actually want for cost, because it pays for the traces that answer production questions and discards the ones that just confirm everything is fine. The catch is that tail sampling has to happen somewhere that sees the whole trace, which is why it usually lives in a collector in front of Datadog rather than in the per-service agent config. More on that placement below, because it is the same place the durable cost fix lives.

Stop indexing logs that you never query

Datadog separates log ingestion from log indexing, and the cost mistake is indexing streams nobody ever searches. A clean way to decide where a stream belongs:

High query frequency, recent: index normally. Application errors, security-relevant events, anything an on-call engineer greps during an incident.
Rarely queried, occasionally needed: route to Flex Logs. Older application logs, audit trails you touch monthly.
Compliance or “just in case,” effectively never queried: archive to your own object storage (S3, GCS) and rehydrate on demand. You pay storage rates, not indexing rates.
Pure noise: drop at ingestion with an exclusion filter and never pay for it at all. Health checks, 200-status access logs, debug output from stable services.

The first move is identifying the drop-at-ingestion category, because it is free savings. A health-check endpoint logging every probe can be a meaningful share of log volume on its own, and no engineer has ever searched for it. Exclusion filters and ingestion-side sampling handle the noise tier; the routing tiers handle everything else. The point is that “reduce log costs” is really a routing decision per stream, not a single retention setting you turn down and hope.

Why the levers stop working

Every lever above is a manual action taken against a continuous, growing flow, and the decay is structural, not a discipline problem.

Run the cardinality script in March, find the metric tagged with pod_name, drop the tag, and bring spend down. In April, a different team ships a service that tags a metric with user_id, because there is nothing between a developer adding a tag and Datadog billing you for it. No admission control, no review, no gate. You pay for a full month of it before your next audit even surfaces it. Then you drop it, and in May, someone adds request_id. The treadmill speeds up as the engineering org grows because cost optimization remains centralized on a single platform team, while telemetry generation is decentralized across everyone who ships code.

The decision about what to emit and how to shape it lives at the source, with hundreds of developers, and the decision about what it costs lives at the destination, with Datadog’s meter. There is nothing in between governing the flow. Static exclusion filters help, but they only act on what you already thought to filter, and you maintain them by hand. Teams that treat Datadog cost as a periodic cleanup never get off the treadmill because the thing producing the cost never stops and is never brought under control.

Coding agents are making the problem worse

Everything above assumed the telemetry was written by humans, shipping at human speed, with at least the occasional engineer who pauses and thinks, “Should this metric really be tagged by pod?”

Coding agents write and ship instrumentation at machine speed, and they make the cardinality problem worse on three fronts.

First, volume. An agent can scaffold a new service, wire up the Datadog tracer, and add metrics in minutes, and a platform team can now stand up more services in a sprint than they used to in a quarter. Every one of those services emits telemetry. The thing producing your bill is no longer gated by how fast humans can write code.

Second, the review brake is gone. The whole structural argument here is that there is nothing between a developer adding a tag and Datadog billing you for it. With a human, there was at least a soft brake: a code reviewer who occasionally caught an obviously expensive tag, an engineer who had been burned by a bill before, and flagged it. Agents generate the instrumentation and frequently open the PR, and the human reviewing it often focuses on logic and correctness, not cardinality. Rarely are we doing observability cost math in a diff review, least of all on telemetry an agent added as a sensible-looking default.

Third, and worst, agents copy patterns. An agent reaches for what looks idiomatic, and a lot of example instrumentation on the internet tags liberally for “observability.” Ask an agent to add metrics to a request handler, and you will often get exactly the catastrophic pattern from the top of this article: latency tagged by endpoint, by status, and by some per-request or per-pod identifier, because that is what the training data shows as thorough instrumentation. The agent has no concept that pod_name multiplies the bill by two hundred. It is optimizing for “well-instrumented,” and “well-instrumented” and “expensive” look identical in code.

For example, imagine a team uses an agent to migrate twelve services onto a shared metrics helper. The helper, also agent-suggested, attaches a standard tag set including a request_id for “traceability.” Each service was a few thousand timeseries before. After the migration, request_id pushes each one into the hundreds of thousands. No human decided to spend that money. No reviewer saw a number that looked wrong; the diff just looked like good, consistent instrumentation. The bill arrives a month later, and the cardinality audit script from earlier now has twelve new offenders to find instead of one.

The manual treadmill is not just slow; it is losing ground. Your audit cadence is fixed at human speed, while telemetry generation has moved to agent speed. You cannot review your way out of it because the volume of instrumentation being shipped has decoupled from the number of humans who could inspect it for cost.

Controlling costs in the pipeline

The durable fix is to add a control layer between your telemetry sources and Datadog so the shaping happens continuously and automatically, not during a quarterly audit. The solution isn’t a manual pipeline, but an intelligent telemetry pipeline that sits in front of the backend and surfaces issues automatically, governing what reaches Datadog, while aggregating high-cardinality metrics down before they become billable timeseries, applying sampling, dropping noisy logs, and routing archival data to cheap storage instead of indexing.

This is lifecycle versus cleanup. Instead of discovering a user_id tag a month after you started paying for it, the pipeline aggregates that cardinality out as the data flows, before it ever hits the meter. Instead of maintaining a growing list of exclusion filters by hand, the control layer enforces shaping policy continuously across every source. And because tail sampling and metric aggregation both need a vantage point that sees the full flow, the pipeline is the natural place they live anyway. The platform team gets governance over the flow without policing every developer’s tagging after the fact.

It also coexists with Datadog instead of replacing it. You keep the dashboards, monitors, and workflows your engineers already depend on. The change is that the volume reaching Datadog is shaped by what you actually use, so cost tracks value rather than raw emissions.

Datadog cost optimization with agentic telemetry management

Sawmills is an agentic telemetry management platform that operates the pipeline between your sources and tools like Datadog. Rather than handing you another dashboard of recommendations to action by hand, Sawmills operates on the telemetry directly: it profiles what you are sending, identifies the cardinality and noise driving cost, and shapes the flow continuously, so the cardinality audit and the exclusion filters and the sampling rules you would otherwise maintain by hand become a standing control that does not decay between quarters, and does not break when an agent ships twelve services overnight.

Because it sits in front of Datadog and coexists with it, getting spend under control does not require you to sacrifice the tooling your engineers rely on. The metrics and logs that carry a real signal reach Datadog intact. The volume that was only ever inflating the bill gets shaped out before it bills. Cost stops being a quarterly fire drill and becomes a property of the system.

Our customers find that Sawmills is ROI positive within 30 days. Schedule a demo to see how agentic telemetry management can help you optimize your Datadog cost.