Logging and monitoring costs: the bill nobody planned for
Observability, meaning logs, metrics and traces, is essential and easy to overspend on. Because it grows with traffic and with how chatty your code is, it can end up costing as much as the servers it watches.
What is billed
Logs. Typically three charges: ingestion, per gigabyte of log data sent in, storage for the retention period and queries or scans when you search them. Ingestion is often the largest.
Metrics. Billed per custom metric per month, or per data point, with a free allowance of the vendor’s own standard metrics. High-resolution metrics and many unique label combinations raise the count quickly.
Traces. Billed per span or per amount ingested, often with sampling to control volume.
Dashboards and alarms. Billed per dashboard and per alarm in some services.
The products are AWS CloudWatch, Azure Monitor with Log Analytics, Google Cloud Logging and Monitoring, and Oracle Logging and Monitoring. Many teams use a third-party platform instead, billed by volume or by host.
Why it grows
- Debug logging left on in production. It can multiply volume tenfold.
- Logging whole requests and responses.
- High-cardinality metrics, such as one label per user or request id, creating millions of series.
- Long default retention. Logs often keep forever unless you set a period.
- Duplicate shipping. The same line sent to two services.
- Audit and flow logs for networks and storage, which can be very large.
- Scale. Ten times the servers means ten times the logs.
How to keep it in check
- Measure first. Find which log groups and metrics cost the most. The top few are usually most of the bill.
- Set a log level policy. Information and above in production, debug only when investigating, and turn it off after.
- Set retention on every log group. Thirty days is enough for most operational logs. Archive what compliance needs to cheap storage. See object storage classes and lifecycle.
- Filter before ingestion. Drop health checks and noisy lines at the source or at the agent.
- Sample traces. Keep a fraction of normal traffic, and keep all errors.
- Control metric labels. Do not put unbounded values in them.
- Aggregate before storing where raw detail is not needed.
- Alarm on log volume. A sudden rise often signals a bug.
- Query with care, narrowing the time range and fields, on services that bill by scan.
Compliance logs
Some rules require keeping audit logs for a fixed period. Keep those, in the cheapest storage that meets the rule, and do not apply that retention to everything.
Where to look on the bill
Group last month’s cost by service and look for the monitoring and logging lines. If they are among the top five, they deserve an owner and a budget. See how to read a cloud bill and the list of other common causes in cloud cost mistakes.
In our estimates
Our estimates do not include logs and monitoring, because the volume depends on your code. Add a line for them from your own measurements. Backup copies are another data set with their own retention, covered in backup and retention costs.