Stream processing costs: Azure Stream Analytics vs Managed Flink vs Dataflow
Stream processing is what turns raw telemetry into something useful while it is still fresh: a vibration alert within seconds, a 5-minute average per machine, a count of rejected parts per shift. Unlike a database query, a streaming job never stops, so you pay for it every hour of the month.
Azure Stream Analytics: streaming units
Stream Analytics jobs are written in a SQL-like language and sized in streaming units (SUs). You pay per SU per hour, and on the Standard V2 plan the hourly price steps down as your monthly SU-hours grow, so larger or longer-running deployments pay less per unit.
- Simple to size: one number, the SU count, which you raise when a job falls behind.
- SQL only for most jobs (with JavaScript or C# functions for extras). That’s quick for filters and windowed aggregates, but limiting for complex logic.
Amazon Managed Service for Apache Flink: KPUs plus extras
AWS runs Apache Flink for you, sized in Kinesis Processing Units (KPUs). One KPU is 1 vCPU and 4 GB of memory, charged per hour. Two extras are easy to miss:
- One additional KPU per application for orchestration, charged whether your job is busy or not. A small job using one KPU actually pays for two.
- Running application storage, 50 GB per KPU, charged per GB-month.
Flink is very capable (Java, Python or SQL, exactly-once processing, large state), which makes it the strongest choice for complex jobs, but the per-application overhead makes many small jobs comparatively expensive. Combining related logic into fewer applications helps.
Google Cloud Dataflow (streaming): vCPU, memory and data processed
Dataflow runs Apache Beam pipelines. For streaming jobs you pay:
- Per vCPU-hour and per GB-hour of memory for the worker machines;
- Streaming Engine data processed, per GB, for the shuffle and state work Google runs outside your workers.
Dataflow autoscales workers with load, which helps spiky traffic, but a streaming job always keeps at least one worker running.
The minimum you always pay
All three charge for the job running, not for the data arriving, so a quiet factory at night costs the same per hour as a busy one at noon. For small workloads, the minimum job size usually decides the bill:
| Service | Smallest always-on job |
|---|---|
| Azure Stream Analytics | The smallest SU size, all month |
| AWS Managed Flink | 1 KPU plus 1 orchestration KPU, plus storage |
| Google Dataflow | One worker’s vCPUs and memory, plus data processed |
Do you need it at all?
Not always. If you only need dashboards and alerts that can be a minute or two behind, the time-series database can often compute rolling averages itself, and the stream processing line disappears. That’s why our estimator lets you switch stream processing off and see the difference.
Edge processing is another option: filtering and aggregating on the factory floor before data reaches the cloud. See edge runtimes compared.