Whole Source or Change Data Capture (CDC)

You normally extract the whole source data (all file records or all rows of database tables) if the data volume is manageable. This mode is suitable for extraction of reference type data sources, such as postal codes. A reference type data is often the data source for dimension tables.

You may have to extract changes of the source data (only the new and changed data since the last extraction) if the volume of the data source is high and extracting the whole data is inefficient or impossible. This mode of data extraction, known as Change Data Capture (CDC), is usually applied to extract operational transaction data, such as sales orders.

In the following section I explain how push-by-source CDC works on the sales order data extraction. Chapter 8, “Regular Population,” covers the other modes.