ETL pipeline costs: Glue vs Data Factory vs Dataflow

By Rohit Athaide · Published 4 Nov 2026

Short answer: an ETL or ELT pipeline extracts data from sources, changes it and loads it somewhere you can query it. Managed services bill for the compute that runs the work and, in some cases, for orchestration. Costs leak through over-sized workers, frequent small runs and full reloads of data that has not changed.

Key points

• Compute time is the main cost, billed per second or minute with minimums.
• Some services also bill per activity run.
• Incremental loads are the biggest saving.
• Many small files slow jobs and raise request charges.
• Match code-first services with code-first services.

Which services are there?

They are equivalent as managed tools for running transforms on a schedule or on events. Some are code-first (you write Spark or Beam jobs) and some are visual.

How do they bill?

1. Compute time. Billed per second or minute for workers, in units such as data processing units or vCPU hours. This is the main cost.
2. Orchestration. Some services bill per pipeline activity run or per trigger.
3. Data movement. Copying between sources may bill by integration runtime hours.
4. Storage and transfer. The data lake and any cross-region copies are separate. See data lake cost basics.

Minimum billing periods matter. A job that runs for ten seconds may be billed for a full minute.

What does a worked example look like?

Made-up round numbers to show the method, not real prices. A job runs every 15 minutes. Each run uses 10 workers with a 1-minute minimum.

Batching four small runs into one hourly run cuts the bill by about 63 percent, because the per-run minimum and startup are paid 730 times instead of 2,920. Incremental loading does not help when the minimum already dominates, which shows why both run frequency and data read matter. If each full reload read 100 times more data, incremental loading would be the bigger saving.

Where do costs leak?

• Over-sized workers. Jobs given more capacity than they use.
• Frequent small runs. Each pays startup time and minimums. Batch them.
• Rewriting everything every run. Process only new or changed data.
• Many small files. They slow jobs and raise request charges.
• Failed jobs that retry without limits.
• Development clusters left running. See dev and test costs.

How can I cut the cost?

1. Run incremental loads.
2. Write columnar, partitioned files. See partitioning and Parquet.
3. Use autoscaling and set a maximum.
4. Use interruptible capacity where the service allows it.

Which mistakes are common?

• Scheduling hourly jobs that could run daily.
• Leaving a development endpoint running over a weekend.
• Reloading full tables for a few changed rows.
• Producing thousands of tiny output files.

Decision checklist

• How often does the data need to be fresh?
• Does each run read only new data?
• Is there a worker maximum and a retry limit?
• Are outputs columnar and partitioned?

Next steps

Our data platform estimator prices a batch transform on each cloud from published rates. For continuous data, see stream processing costs, and for the destination see data warehouse pricing.