← all topics

Data Engineering

Data Warehouse

Columnar storage and separated compute. Why a scan of a billion rows can cost cents or hundreds of dollars.

Levels foundation / engineer / advanced
Depth 2
Time 5h
Kind system
On Data Engineer

Grasp

A warehouse stores values down the column rather than across the row, and that single choice explains most of its behaviour. Summing one column reads one column, not every byte of every record. Values of the same type, sitting together and often repeating, compress far harder than a mixed row does. This is why a query that would crawl on a transactional database returns in seconds here, and why the same engine is a poor choice for writing one row at a time.

The second choice is that compute is separate from storage. Data sits in cheap object storage; query engines are spun up against it and billed for what they read. Storage costs almost nothing, so the bill is made almost entirely of scans.

That is the part worth internalising, because it makes cost a property of the query and the table layout rather than of the data itself. The same billion-row table can answer one question for cents and another for a fortune, depending on whether the engine could skip most of it. Partitioning and clustering exist to make skipping possible; a query with no filter on the partitioning column cannot skip anything. Most warehouse bills are not caused by large data. They are caused by queries that read all of it.