Observability doesn’t stop at traces and metrics — logs complete the trifecta. If you’re exploring how to manage log data using OpenTelemetry Collector, you’re in the right place. This guide walks through everything a DevOps engineer needs to know about OpenTelemetry Collector logs: what they are, how they work, how to configure them, and how to manage them in production.
OpenTelemetry Collectors and Logs
The OpenTelemetry Collector is a vendor-agnostic service for receiving, processing, and exporting telemetry data. While it's often used for metrics and traces, it's increasingly powerful for handling logs.
Key functions include:
- Receivers to ingest logs from sources like FluentBit, file logs, or syslog
- Processors to modify, batch, or filter logs
- Exporters to send logs to storage backends or observability tools
OpenTelemetry supports two core deployment patterns: the agent (running as a sidecar or DaemonSet) and the gateway (a centralized collector). In practice, teams often combine both, using agents to gather local data and a gateway to handle processing and exporting. This separation allows you to minimize footprint on app nodes while retaining centralized control.
Collector architecture should be driven by scale and latency needs. Gateways can handle transformation, enrichment, and routing across teams, while agents ensure minimal local disruption. For hybrid clouds, edge locations, or high-SLA environments, using layered collector topologies is becoming increasingly common.
Example: Filelog Receiver + OTLP Exporter
receivers:
filelog:
include: [/var/log/*.log]
exporters:
otlp:
endpoint: sawmills-otel-endpoint:4317
service:
pipelines:
logs:
receivers: [filelog]
exporters: [otlp]Types of OpenTelemetry Logs
OpenTelemetry supports structured logs, typically emitted as JSON, which allows better indexing and querying.
Common formats include:
- OTLP Logs: Native format used by OTel's protocol
- Syslog: Widely used in traditional servers
- Filelog: Tail logs from local or mounted files
- HTTP JSON: From custom or third-party agents
These logs carry context through trace and span IDs, enabling correlation with metrics and traces in a unified observability platform.
Methods of Data and Log Collection in OpenTelemetry
Logs can be ingested into the OTel Collector via several receivers:
Supported Receivers:
- filelog: Tail application log files
- syslog: Capture network-based logs
- fluentforward: Integrate with FluentBit/Fluentd
- otlp: Accept logs from SDKs or agents over gRPC or HTTP
You can combine receivers to capture data from legacy systems (via syslog), container workloads (via filelog), and modern SDK-based services (via OTLP) all in one place.
To reduce ingestion errors and improve performance, production-grade collectors often include retry and timeout processors. You can also configure conditional routing using the routing processor to send different log types to different pipelines based on attributes like severity, source, or environment.
Advanced collectors use attribute-based routing to isolate verbose or noisy logs, reduce downstream pressure, and ensure high-SLA logs always reach their destination. Many teams also enrich logs with metadata (cloud.region, pod.name, environment) using the attributes processor.
Storing and Exporting OpenTelemetry Logs
After logs are collected and optionally processed, they need to be exported.
Exporter Options:
- otlp for OpenTelemetry backends
- logging for console/debug output
- loki, splunk_hec, elasticsearch for specialized storage
Example exporter config to Loki:
exporters:
loki:
endpoint: http://loki:3100/api/prom/push
labels:
job: my-app
Batching logs before export improves efficiency:
processors:
batch:
timeout: 5s
send_batch_size: 100
You can also use the attributes processor to rename or sanitize fields before sending them downstream, which is especially useful when forwarding logs to security-sensitive storage layers.
For multitenant environments, use the resource processor to attach team or app-specific metadata, making it easier to filter logs downstream in systems like Grafana Loki or Splunk.
OpenTelemetry Logging Best Practices
- Enable trace context in your logs to improve correlation with traces.
- Use structured logs (preferably JSON) for better indexing and querying.
- Batch logs before exporting to reduce network overhead.
- Set retention rules on your log storage platform.
- Apply processors to redact sensitive data or normalize formats.
- Route logs by source or severity using the routing processor to keep pipelines focused and performant.
- Validate configurations in CI/CD to prevent broken telemetry after deployments.
- Avoid shipping every log line: reduce verbosity in dev and test environments using filter processors.
- Implement dead-letter queues or fallback exporters to avoid silent data loss.
A new way to manage telemetry data
OpenTelemetry Collector gives DevOps engineers granular control over how logs are ingested, processed, and exported — but it requires thoughtful configuration. By using structured logging, trace context, and strategic exporters, you can unify telemetry across your stack.
Want even more control over your teleemtry data? Sawmills brings AI-powered telemetry management — giving you real-time visibility, smart recommendations, and in-stream control over your data. From reducing log volume and enforcing formatting standards to managing collector pipelines and preventing cardinality spikes, Sawmills helps you optimize your telemetry before it hits your backend. Explore cost impact, enrich or drop data, and automate compliance — all from a single interface built on OpenTelemetry. Schedule a demo to see how Sawmills delivers smarter, faster observability.
