Scenario Detection Stage
Submitted by @stodoroska from Interworks Based on the source data, we are creating action flags that later will be used in the Action stage pipeline. Screenshot of pipeline Configuration Because we are working with a large scale of data, the target records are grouped in the chunks of 2000 records. This makes the pipeline faster and less memory is used. Sources: CSV file Targets: CSV file and PGP-encrypted file Snaps used: Mapper, Join, Group by N, Splitter, CSV Parser, File Writer, File Reader, Union, Copy, PGP Encrypt Downloads Attach pipelines and any necessary resources such as expression libraries or source files.
