Reusable pipeline with Aggregate / Group by fields
Is it possible (and how) to build reusable pipelines with Aggregate / Group by fields, where the “group by” clauses are dynamic (and the number of clauses is provided as a list in the document).
I’ve been trying this with something as follows, but it doesn’t seem to be working.
The premise is that $counter contains a list of fields to group by (obviously the length of the list is unknown, and can change.
for example
[
"FieldA", "FieldB", ...
]
Maybe there’s another way to perform this type of aggregation - but i’m currently stuck (maybe also just out of ideas for now) on how to perform this type of thing.
I need to build this to detect unique values in a dataset (unique based on a subset of columns).
currently, i need to build a seperate pipeline / pipeline segment for each table - allowing to specify this as parameter would greatly simplify our pipelines.
Hi Matthias,
Good day, what I did before was put the streaming documents in an array (you can use the Gate snap) then loop through the array doing the aggregate and grouping
See attached poc pipeline
input:
output:
test_2021_11_10.slp (13.4 KB)
Thanks,
EmEm