Reusable pipeline with Aggregate / Group by fields

[Matthias]

Is it possible (and how) to build reusable pipelines with Aggregate / Group by fields, where the “group by” clauses are dynamic (and the number of clauses is provided as a list in the document).
I’ve been trying this with something as follows, but it doesn’t seem to be working.

The premise is that $counter contains a list of fields to group by (obviously the length of the list is unknown, and can change.

for example

[
    "FieldA", "FieldB", ...
]

Maybe there’s another way to perform this type of aggregation - but i’m currently stuck (maybe also just out of ideas for now) on how to perform this type of thing.

I need to build this to detect unique values in a dataset (unique based on a subset of columns).
currently, i need to build a seperate pipeline / pipeline segment for each table - allowing to specify this as parameter would greatly simplify our pipelines.

[alchemiz · Accepted solution]

Hi Matthias,

Good day, what I did before was put the streaming documents in an array (you can use the Gate snap) then loop through the array doing the aggregate and grouping

See attached poc pipeline

input:

output:

test_2021_11_10.slp (13.4 KB)

Thanks,
EmEm

[ptaylor]

Matthias,

I’m afraid the Aggregate snap doesn’t have the flexibility you’re looking for. While each field name in each row of both fields tables can be dynamically derived using an expression, you can’t dynamically populate the entire content of these tables.