Getting Started with AWS and OTel Collectors

‍What Is AWS?

Amazon Web Services (AWS) is the leading cloud computing platform, offering scalable infrastructure and services for storage, compute, networking, and more. It powers applications for startups and enterprises alike, providing the backbone for modern cloud-native development.

What Is OpenTelemetry?

OpenTelemetry (often abbreviated as OTel) is an open-source observability framework designed to standardize the collection of telemetry data such as logs, metrics, and traces. It enables developers and DevOps teams to collect, process, and export observability data from their applications to various backends.

At the heart of the OpenTelemetry ecosystem is the OTel Collector. The collector is a vendor-agnostic agent that receives telemetry data, processes it (e.g., filtering, batching, transforming), and exports it to the observability platform of your choice (e.g., Amazon CloudWatch, X-Ray, Prometheus, or third-party services like Datadog).

How Do AWS and OpenTelemetry Work Together?

AWS provides a distribution of the OpenTelemetry Collector known as the AWS Distro for OpenTelemetry (ADOT). It’s an AWS-supported version of the OTel Collector, preconfigured to integrate seamlessly with AWS services like:

• Amazon CloudWatch (for metrics and logs
• AWS X-Ray (for distributed traces)
• Amazon ECS, EKS, and EC2 (for infrastructure observability)

This combination allows you to instrument your applications using OpenTelemetry SDKs and forward the telemetry data through the AWS OTEL Collector to native AWS services, offering deep visibility into system performance and behavior.

Getting Started Using OTel Collectors and AWS

Step 1: Choose Your Environment

Before you begin, decide where you want to run the AWS OTEL Collector:

• Amazon EC2
• Amazon ECS (Fargate or EC2 launch type)
• Amazon EKS (Kubernetes)

For this guide, we’ll use Amazon EKS as an example.

Step 2: Install the AWS OTEL Collector on EKS

Step 2.1: Clone the ADOT Configuration Repository

git clone https://github.com/aws-observability/aws-otel-collector.git
cd aws-otel-collector/examples/k8s/

Step 2.2: Customize the Configuration (Optional)

Edit the otel-config.yaml to configure which data sources (receivers), processors, and exporters you want to use. For example, to collect metrics and traces and export to CloudWatch and X-Ray:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  awscloudwatch:
  awsxray:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      exporters: [awscloudwatch]

‍

Step 2.3: Deploy the Collector to EKS

Apply the Kubernetes manifests to your cluster:

kubectl apply -f adot-daemonset.yaml

This deploys the OTEL Collector as a DaemonSet, ensuring it runs on every node.

Step 3: Instrument Your Application

Install the OpenTelemetry SDK for your language (Java, Python, Node.js, etc.).

Example for Python:

pip install opentelemetry-sdk opentelemetry-exporter-otlp

Configure your app to export data to the OTEL Collector via OTLP:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

Step 4: View Your Data in AWS

• Traces: View in AWS X-Ray console
• Metrics: View in Amazon CloudWatch under custom namespaces
• Logs: If logs are collected, they’ll also appear in CloudWatch Logs

Best Practices for Using OpenTelemetry and AWS

• Use Resource Attributes: Tag telemetry data with metadata (e.g., service name, environment) to aid filtering and analysis.
• Minimize Data Volume: Use processors to batch and drop unneeded data before export.
• Secure Communication: Use TLS between your app and the OTEL Collector.
• Centralize Configuration: Use shared config files or Kubernetes ConfigMaps for easier management.
• Monitor the Collector: Enable internal metrics and health checks to ensure your collector is working properly.

Conclusion

The AWS OTEL Collector bridges the power of OpenTelemetry with the scalability of AWS, enabling developers to collect and export telemetry data seamlessly. By installing the collector on AWS infrastructure like EKS, ECS, or EC2, and instrumenting your applications with the OTel SDK, you gain deep visibility into logs, metrics, and traces. Whether you’re debugging distributed applications or fine-tuning performance, the AWS OTEL Collector is a foundational tool in your observability toolkit.

Looking for more ways to manage and optimize your telemetry pipeline?

‍ Schedule a demo with Sawmills →

‍