Container Logs in Kubernetes: How to View and Collect Them

In today's cloud-native ecosystem, Kubernetes has become the de facto standard for container orchestration. As organizations scale their microservices architecture and embrace DevOps practices, the ability to effectively monitor and troubleshoot containerized applications becomes paramount. Container logs serve as the primary source of truth for understanding application behavior, debugging issues, and maintaining observability across your distributed systems.

Whether you're a DevOps engineer, SRE, or infrastructure specialist, understanding how to view and collect container logs in Kubernetes is essential for maintaining robust, production-ready applications. This comprehensive guide will walk you through everything you need to know about container logging in Kubernetes, from basic commands to advanced collection strategies.

Installing kubectl

Before you can view and collect container logs, you'll need to install kubectl, the Kubernetes command-line tool. Here's how to install it on different operating systems:

Installing kubectl on macOS

Using Homebrew (recommended):

brew install kubectl

Using curl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Installing kubectl on Windows

Using Chocolatey:

choco install kubernetes-cli

Using winget:

winget install -e --id Kubernetes.kubectl

Using curl:

curl.exe -LO "https://dl.k8s.io/release/v1.28.0/bin/windows/amd64/kubectl.exe"

Verifying Your Installation

After installation, verify that kubectl is working correctly:

kubectl version --client

You should see output similar to:

Configuring kubectl

Once installed, you'll need to configure kubectl to connect to your Kubernetes cluster:

# If using a kubeconfig file
export KUBECONFIG=/path/to/your/kubeconfig

# Or copy the kubeconfig file to the default location
cp /path/to/your/kubeconfig ~/.kube/config

# Test the connection
kubectl cluster-info

What Are Container Logs?

Container logs are the output streams generated by applications running inside containers. These logs typically include:

• Application logs: Standard output (stdout) and standard error (stderr) from your applications
• System logs: Operating system and runtime information
• Access logs: HTTP requests, database queries, and API calls
• Error logs: Exceptions, stack traces, and debugging information

In Kubernetes, each container generates logs that are captured by the container runtime (like containerd or Docker) and made available through the Kubernetes API. These logs are crucial for:

• Observability: Understanding what's happening inside your containers
• Debugging: Identifying and resolving issues quickly
• Monitoring: Tracking application performance and health
• Compliance: Meeting audit and regulatory requirements

How to View Container Logs in Kubernetes

Method 1: Using kubectl logs (Most Common)

The kubectl logs command is the primary tool for viewing container logs in Kubernetes. Here's how to use it effectively:

Basic Log Viewing

# View logs from a specific pod
kubectl logs <pod-name>

# View logs from a specific container in a multi-container pod
kubectl logs <pod-name> -c <container-name>

# Follow logs in real-time (like tail -f)
kubectl logs -f <pod-name>

# Show timestamps with logs
kubectl logs --timestamps <pod-name>

Advanced Log Viewing Options

# View logs from the last 100 lines
kubectl logs --tail=100 <pod-name>

# View logs from the last hour
kubectl logs --since=1h <pod-name>

# View logs from a specific time
kubectl logs --since-time="2024-01-15T10:00:00Z" <pod-name>

# View logs from previous container instance (if pod restarted)
kubectl logs --previous <pod-name>

Method 2: Using kubectl exec for Direct Access

For more detailed investigation, you can exec into a container and view logs directly:

# Exec into a container
kubectl exec -it <pod-name> -- /bin/bash

# Once inside, you can view log files directly
cat /var/log/application.log
tail -f /var/log/application.log

Method 3: Using Kubernetes Dashboard

If you have the Kubernetes Dashboard installed, you can view logs through the web interface:

1. Navigate to your Kubernetes Dashboard
2. Go to the "Pods" section
3. Click on the specific pod
4. Click on the "Logs" tab to view container logs

How to Collect Container Logs

While viewing logs is useful for debugging, collecting and centralizing logs is essential for production environments. Here are several approaches:

1. Using kubectl for Log Collection

# Save logs to a file
kubectl logs <pod-name> > pod-logs.txt

# Collect logs from multiple pods
kubectl logs -l app=myapp > all-app-logs.txt

# Collect logs from all pods in a namespace
kubectl logs --all-containers=true -n <namespace> > namespace-logs.txt

2. Using kubectl for Advanced Log Collection

kubectl provides powerful options for collecting logs from multiple sources:

# Collect logs from all pods with a specific label
kubectl logs -l app=myapp --all-containers=true

# Collect logs from all pods in a namespace
kubectl logs --all-containers=true -n <namespace>

# Collect logs from a specific time range
kubectl logs --since=2h <pod-name>

# Collect logs and save with timestamps
kubectl logs --timestamps <pod-name> > logs-with-timestamps.txt

# Collect logs from multiple pods simultaneously
for pod in $(kubectl get pods -l app=myapp -o name); do
  kubectl logs $pod > $(echo $pod | cut -d'/' -f2)-logs.txt
done

3. Using kubectl for Log Analysis

kubectl can be combined with standard Unix tools for powerful log analysis:

# Search for specific patterns in logs
kubectl logs <pod-name> | grep "ERROR"

# Count log entries by level
kubectl logs <pod-name> | grep -o "level=[A-Z]*" | sort | uniq -c

# Extract timestamps and analyze log frequency
kubectl logs --timestamps <pod-name> | awk '{print $1}' | sort | uniq -c

# Monitor logs in real-time across multiple pods
kubectl logs -f -l app=myapp --all-containers=true

Best Practices for Container Logging

1. Structured Logging

Use structured logging formats like JSON to make logs more searchable and analyzable:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "INFO",
  "message": "User login successful",
  "user_id": "12345",
  "ip_address": "192.168.1.100",
  "service": "auth-service"
}

2. Log Levels and Severity

Implement proper log levels to filter and prioritize logs:

• DEBUG: Detailed information for debugging
• INFO: General information about application flow
• WARN: Warning messages for potential issues
• ERROR: Error conditions that need attention
• FATAL: Critical errors that may cause application failure

3. Log Rotation and Retention

Configure log rotation to prevent disk space issues:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logrotate-config
data:
  logrotate.conf: |
    /var/log/containers/*.log {
      daily
      rotate 7
      compress
      missingok
      notifempty
    }

4. Resource Limits for Logging

Set appropriate resource limits for logging containers:

resources:
  requests:
    memory: "64Mi"
    cpu: "50m"
  limits:
    memory: "128Mi"
    cpu: "100m"

Troubleshooting Common Logging Issues

Issue 1: No Logs Appearing

Symptoms: kubectl logs returns empty or no output

Solutions:

# Check if the pod is running
kubectl get pods

# Check container status
kubectl describe pod <pod-name>

# Verify log configuration
kubectl exec <pod-name> -- ls -la /var/log/

Issue 2: Logs Not Being Collected by kubectl

Symptoms: kubectl logs returns empty or partial results

Solutions:

# Check if the pod is running and ready
kubectl get pods -o wide

# Verify the container is actually generating logs
kubectl exec <pod-name> -- ps aux

# Check if logs are being written to stdout/stderr
kubectl exec <pod-name> -- ls -la /proc/1/fd/

# Verify log configuration in the container
kubectl describe pod <pod-name>

Issue 3: High Log Volume Impacting Performance

Symptoms: Cluster performance degradation due to excessive logging

Solutions:

• Implement log filtering and sampling
• Use log buffering and batching
• Configure appropriate resource limits
• Consider using log compression

Advanced Logging Strategies

1. Multi-Tenant Logging

For organizations with multiple teams or projects:

apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    team: team-a
    environment: production

2. Log Encryption and Security

Implement log encryption for sensitive data:

apiVersion: v1
kind: Secret
metadata:
  name: logging-credentials
type: Opaque
data:
  username: <base64-encoded-username>
  password: <base64-encoded-password>

3. Real-Time Log Streaming

For real-time monitoring and alerting:

# Stream logs to external monitoring system
kubectl logs -f <pod-name> | tee /tmp/stream.log | nc monitoring-host 514

Advanced Log Analysis with kubectl

Setting Up Log-Based Monitoring

You can use kubectl in combination with shell scripts to create simple log monitoring:

#!/bin/bash
# Monitor for error patterns in logs
while true; do
  if kubectl logs <pod-name> --since=5m | grep -q "ERROR"; then
    echo "$(date): Error detected in logs"
    # Send notification or trigger alert
  fi
  sleep 60
done

Conclusion

Effective container log management in Kubernetes is crucial for maintaining observability and ensuring smooth operations in your cloud-native environment. By implementing the techniques and best practices outlined in this guide, you'll be able to:

• View and collect logs efficiently using kubectl and other tools
• Implement centralized log aggregation for better visibility
• Troubleshoot issues quickly with proper log analysis
• Scale your logging infrastructure as your applications grow

Remember that logging is not just about collecting data—it's about gaining insights into your application's behavior and maintaining operational excellence in your Kubernetes clusters.

Need help implementing robust logging solutions for your Kubernetes environment? Sawmills specializes in cloud-native infrastructure and can help you design and implement comprehensive logging strategies that scale with your applications. Our team of DevOps experts can assist with everything from basic log collection to enterprise-grade observability solutions.

Additional Resources

• Kubernetes Logging Architecture
• kubectl logs Documentation
• Kubernetes Troubleshooting Guide
• Container Logging Best Practices