Datadog Log Ingestion & APM Bill Shock Calculator

# datadog_exclusion_filter.tf - Stop indexing non-critical logs to save 75%
resource "datadog_logs_index" "production_index" {
  name           = "main"
  filter {
    query = "env:production"
  }
  
  # EXCLUSION FILTER: Ingest but DO NOT index verbose HTTP access and healthcheck logs
  exclusion_filter {
    name       = "exclude_healthchecks_and_debug"
    is_enabled = true
    filter {
      query        = "status:info OR status:debug OR http.url:\"/healthz\" OR user_agent:\"ELB-HealthChecker\""
      sample_rate  = 0.05 # Index only 5% sample for traffic trends; drops 95% of indexing fees!
    }
  }
}

Datadog Log Ingestion Architecture: Ingest vs Index Split & Bill Shock Mitigation

Datadog's logging pricing is partitioned into two distinct charges: Ingestion ($0.10 per GB) and Indexing ($1.06 to $2.55 per million log events depending on retention window from 3 to 30 days). In high-volume Kubernetes microservice clusters and LLM production gateways, logging verbose API request/response payloads or high-frequency health-check pings can cause monthly Datadog bills to spike by tens of thousands of dollars without warning.

Production Pipeline: Vector (Datadog Aggregator) Edge Redaction & Sampling

Deploying an open-source Vector sidecar on Kubernetes clusters pre-filters high-volume HTTP 200 health probes and scrubs sensitive PII before logs hit the Datadog intake endpoint:

# vector.yaml: Pre-Ingestion Filter & Sampling Pipeline
sources:
  kubernetes_logs:
    type: kubernetes_logs

transforms:
  filter_health_checks:
    type: filter
    inputs: [kubernetes_logs]
    # Drop 99% of successful health checks and readiness probes
    condition: '!(.message == "GET /healthz HTTP/1.1" && .status == 200)'

  sample_info_logs:
    type: sample
    inputs: [filter_health_checks]
    rate: 10 # Sample only 1 in 10 INFO logs; retain 100% of WARN/ERROR

sinks:
  datadog:
    type: datadog_logs
    inputs: [sample_info_logs]
    default_api_key: "${DATADOG_API_KEY}"
    compression: gzip

Enterprise Governance & Exclusion Filter Checklist

• Datadog In-App Exclusion Filters: Ingested logs do not have to be indexed. Configure Datadog Log Exclusion Filters to send 95% of routine debug logs straight to S3/GCS archival storage ($0.023/GB) while indexing only error traces.
• Sensitive Data Scanner: Enable Datadog Sensitive Data Scanner to prevent accidental ingestion of customer credit card numbers, JWT tokens, and OpenAI API keys.
• Automated Anomaly Budget Alerts: Set up a daily monitor on `datadog.estimated_usage.logs.ingested_bytes` to trigger PagerDuty alerts whenever daily log volume exceeds 150% of the baseline average.

Frequently Asked Datadog Log Cost Mitigation Questions

Q: What is the single most common cause of sudden Datadog bill spikes?
A: Enabling DEBUG-level logging in high-traffic production Kubernetes services or ingesting high-frequency load balancer health check pings (`/healthz`), which can generate hundreds of gigabytes of unneeded logs daily.

Q: How do In-App Exclusion Filters save money without dropping data?
A: Exclusion filters route matching logs directly to low-cost Cloud Storage / S3 archives ($0.023/GB) for compliance auditing while excluding them from Datadog's expensive $1.70/1M indexing fee.