Reusable pipeline with Aggregate / Group by fields
Is it possible (and how) to build reusable pipelines with Aggregate / Group by fields, where the “group by” clauses are dynamic (and the number of clauses is provided as a list in the document).
I’ve been trying this with something as follows, but it doesn’t seem to be working.
The premise is that $counter contains a list of fields to group by (obviously the length of the list is unknown, and can change.
for example
[
"FieldA", "FieldB", ...
]
Maybe there’s another way to perform this type of aggregation - but i’m currently stuck (maybe also just out of ideas for now) on how to perform this type of thing.
I need to build this to detect unique values in a dataset (unique based on a subset of columns).
currently, i need to build a seperate pipeline / pipeline segment for each table - allowing to specify this as parameter would greatly simplify our pipelines.
@heidi.andrew - here is another pipeline that provides a couple more examples how this could be done. In this pipeline, there are two examples - the first is similar to what @alchemiz has done, but with an added step to simplify the syntax of the record conversion into an array then object. The second path steps it down even farther using all basic snap functionality to split the input stream so it can be parsed separately based on the input row type.
I hope this helps!
Community 15513 - Multi-row CSV_2023_02_22.slp (24.3 KB)
