Forum Discussion

Matthias's avatar
Matthias
New Contributor III
4 years ago
Solved

Reusable pipeline with Aggregate / Group by fields

Is it possible (and how) to build reusable pipelines with Aggregate / Group by fields, where the “group by” clauses are dynamic (and the number of clauses is provided as a list in the document).
I’ve been trying this with something as follows, but it doesn’t seem to be working.

The premise is that $counter contains a list of fields to group by (obviously the length of the list is unknown, and can change.

for example

[
    "FieldA", "FieldB", ...
]

Maybe there’s another way to perform this type of aggregation - but i’m currently stuck (maybe also just out of ideas for now) on how to perform this type of thing.

I need to build this to detect unique values in a dataset (unique based on a subset of columns).
currently, i need to build a seperate pipeline / pipeline segment for each table - allowing to specify this as parameter would greatly simplify our pipelines.