CloudCostCompareData lake cost basics: storage, queries and transforms
A data lake keeps raw and processed data as files in cheap object storage, and lets you query it with a SQL engine that reads the files in place. It is one of the more flexible ways to run analytics, and its costs are different from a database’s.
The four lines
1. Storage. The files sit in object storage, billed per gigabyte per month. It is usually the smallest line and the one people worry about most. See object storage classes and lifecycle.
2. Queries. A serverless SQL engine reads the files when you run a query. Several are billed by the amount of data scanned, not by time. See serverless SQL pay per scan. This is the line that grows with how many people use the lake, and the one most often underestimated.
3. Transformation. Jobs that clean, convert and combine raw files into usable tables, running in batch. They are billed for the compute they use while they run, often in units of processing time. Examples of the services are AWS Glue, Azure Data Factory data flows, Google Dataflow and Oracle Data Integration.
4. Data transfer. Results leaving the cloud, or data moving between regions, is billed as egress. See data transfer costs.
How our estimate splits them
The data platform estimate asks for the amount of data stored, how much each query scans and how many queries you run, how many hours of batch transformation you need, and how much data you send out. It prices each of the four lines on each cloud where an equivalent service exists, and shows the formula behind each line.
One cloud has no service that bills per amount of data scanned. Rather than compare a per-scan service with a per-hour one, the estimate marks that line “Not offered” with the reason, and the total is incomplete. That is the honest result, not a gap in our work.
Why queries surprise people
Storage cost grows with data you keep. Query cost grows with how much data your queries read, and that depends on how the data is laid out more than on how big it is. A query that needs one column of one day can read the whole table if the files are not organized for it. A dashboard that refreshes every few minutes runs the same query all day.
So two lakes with the same stored data can have query bills that differ by ten times. The fixes are in partitioning and Parquet.
Starting out
- Store data in a columnar format, usually Parquet, compressed.
- Partition by the column you filter on most, usually date.
- Keep a small number of larger files rather than many tiny ones.
- Estimate queries as scans, not as counts. Ask how much each query reads.
- Put a budget alert on the query service. See budgets and cost alerts.
- Archive what you rarely read to colder storage, with lifecycle rules.
Lakes and warehouses
A data warehouse stores data in the engine’s own format and is billed for compute capacity or per query, depending on the product. The difference is not only cost: a warehouse is often faster and easier for heavy, frequent BI, while a lake is cheaper for large volumes that are queried lightly. For a lighter comparison of the trade, start with the three-line sizing in the estimator and compare its queries line with your own usage.