BigID dramatically reduces observability bill with Sawmills

About BigID

BigID is redefining how organizations manage data security, privacy, and governance. Founded in 2016, BigID introduced a groundbreaking idea: start with the data itself. As enterprises grappled with growing data volumes, evolving regulations, and the rise of AI, BigID offered a smarter, contextual approach to understanding and managing data.

Since then, BigID has led the industry with firsts: the first identity-aware discovery engine, the first actionable DSPM solution, and the first unified Data Security Platform. Today, BigID helps global organizations discover, classify, and control data everywhere it lives, across cloud, SaaS, unstructured sources, and AI pipelines. With BigID Next, the company continues to empower security and governance teams to reduce risk, ensure compliance, and unlock data value, delivering clarity and trust in an era defined by complexity.

The challenge: A telemetry pipeline with runaway log costs

BigID operates a large-scale observability environment running across multiple regions. As their footprint expanded, so did the volume of logs and metrics streaming into Datadog, along with the cost.

"We were very focused on our cost-efficiency initiatives, and Datadog was a major pain point. We'd tried filters, exclusions, and sampling. But our bill was still too high and needed constant monitoring and handling."

While techniques like sampling helped reduce some of the volume, they weren't enough to control spend at scale. And for many logs, BigID needed better strategies to reduce noise while preserving context and quality.

Centralized, AI-driven telemetry optimization with Sawmills

BigID adopted Sawmills' smart telemetry management platform to gain real-time visibility and control over its entire data stream before it hits the observability backend. Sawmills operates in-stream within BigID environments to manage routing, aggregation, formatting, and volume in a single interface.

To maintain compliance and auditability, low-priority logs weren't discarded but routed to Amazon S3 for rehydration purposes, providing teams with the flexibility to restore logs as needed without incurring additional ingest costs.

BigID then implemented log aggregation using Sawmills to further reduce volume without losing clarity. Aggregation groups similar logs, like repeated status messages, into structured summaries, preserving key context while eliminating redundancy.

Beyond the volume savings, aggregation also made logs more useful. Developers benefited from cleaner, more consistent log formatting, which improved parsing and search. Summarized entries can help teams spot anomalies faster and enable quicker root cause analysis. And with less noise reaching the backend, query performance and indexing costs improve system-wide.

“We moved our existing sampling filters to aggregation because it gave us a smarter way to reduce noise and retain fidelity. It keeps our logs useful for debugging while ensuring we stay within budget.”

^{Avior Malkukian, Senior Director of DevOps @ BigID}

Continuous, In-Stream Telemetry Optimization

BigID activated Sawmills Insights to analyze telemetry data in-stream, alerting the DevOps organization to issues such as duplicate or redundant log lines, excessively verbose debug messages, and more. Each surfaced pattern includes a recommended fix that can be applied in the Sawmills UI, requiring no code changes or engineering effort.

"Sawmills gives us a centralized, intelligent layer to control everything from volume to log quality," said Avior. "We are able to optimize across all clusters without disrupting our engineering teams' workflows."

This capability allows BigID's DevOps team to proactively manage and enforce telemetry standards, continuously optimize their pipeline, and significantly reduce observability overhead. Instead of chasing developers to fix code issues, the DevOps team is empowered to make changes on the fly without deployment.

And with Sawmills built in Livetail feature, BigID can view data on the fly and simulate the impact of Insights recommendations before applying them to their pipeline.

The results - big cost savings and sharper insights from efficient telemetry control

BigID's telemetry transformation delivered immediate, measurable results. DevOps maximizes control over their pipeline, and developers are free to do what they do best: build products. The immediate results are visible in the numbers:

• Sawmills has helped BigID move faster and achieve big savings:
• 81% reduction in log costs, a major cut to BigID's Datadog bill
• 89% reduction in daily telemetry ingest, from 111 TB to 12 TB/day
• 92% reduction in message volume, from 72.8 B to 5.46 B msg/day
• Compliance-ready rehydration with full log access maintain via s3 without the ingest overhead
• Continuous, in-stream optimization and recommendations that can be applied instantly without code deployment

“With Sawmills, we’re in control of our telemetry,” said Avior. “The aggregation processor and automatic insights are just the beginning. We’re excited about the broader roadmap and how it will continue to transform how we manage and scale our observability infrastructure. As a team that’s always pushing the boundaries of what’s possible in DevOps, it’s been great partnering with the Sawmills team. They move fast, understand the space, and are building exactly what modern engineering teams need.”