Object storage for sensor archives: tiers, request costs and retention
Raw sensor readings are worth keeping: for audits, for training models, and for answering questions nobody has asked yet. Keeping them in the time-series database gets expensive fast, so most designs keep a few weeks “hot” in the database and archive everything to object storage: Azure Blob Storage, Amazon S3 or Google Cloud Storage.
Object storage is cheap per gigabyte. What catches people out is the other charges.
The three things you pay for
- Storage, per GB per month. This is the number in every headline price.
- Requests (also called operations or transactions): every write, read and list. Writes cost more than reads.
- Retrieval and early deletion, on the colder tiers only.
Requests: why file size matters
Sensor data arrives as a stream of tiny messages. If you write each message as its own file, you pay a write request per message, and request charges can end up larger than the storage itself.
The fix is to batch readings into bigger files before writing: collect a few minutes of data per device or per line and write one file. Our estimator assumes 4 MB archive files, which keeps request costs small. Columnar formats like Parquet also compress sensor data very well and are faster to query later.
| Write pattern | Effect on cost |
|---|---|
| One file per message | Request charges dominate; many millions of writes a month |
| One file per device per few minutes | Requests become a small share of the bill |
| Compressed Parquet files of several MB | Lowest storage and request cost, easiest to analyse later |
Storage tiers: cheaper to keep, dearer to read
Every cloud has colder tiers that cost much less to store but charge to read, and set a minimum storage duration: delete or move data earlier and you pay for the rest of the period anyway.
| Hot / standard | Infrequent access | Cold | Archive | |
|---|---|---|---|---|
| Azure Blob | Hot | Cool (30-day minimum) | Cold (90 days) | Archive (180 days; offline, hours to restore) |
| Amazon S3 | Standard | Standard-IA (30 days) | Glacier Instant / Flexible Retrieval (90 days) | Glacier Deep Archive (180 days; hours to restore) |
| Google Cloud Storage | Standard | Nearline (30 days) | Coldline (90 days) | Archive (365 days) |
Two more details:
- Small objects are penalised on some colder tiers (for example, S3 Standard-IA bills each object as at least 128 KB). Another reason to write bigger files.
- Retrieval fees on cold tiers mean a one-off reprocessing of a year of data can cost more than storing it did.
A sensible pattern for factory data
- Hot tier for the first month or two, while recent data is still read often.
- Lifecycle rule to move files to a cool or cold tier after that. All three clouds can do this automatically, at no extra cost beyond the transition request.
- Archive tier for data you keep only for compliance.
- Expire data you’ll never need, rather than paying to keep it forever.
Our estimates price the hot tier only, which is the conservative case. With lifecycle rules, long archives cost noticeably less than the estimate shows.
Redundancy
The prices we compare are for single-region storage (for example, Azure LRS). Zone-redundant and geo-redundant options keep extra copies, in other data centres or another region, and cost more. For raw telemetry that can’t be recreated, the extra copy is often worth it.
Compare current prices
- Object storage prices on each cloud
- Estimate your archive by setting how many months to keep