cancel
Showing results for 
Search instead for 
Did you mean: 

Reusable pipeline with Aggregate / Group by fields

Matthias
New Contributor III

Is it possible (and how) to build reusable pipelines with Aggregate / Group by fields, where the “group by” clauses are dynamic (and the number of clauses is provided as a list in the document).
I’ve been trying this with something as follows, but it doesn’t seem to be working.

The premise is that $counter contains a list of fields to group by (obviously the length of the list is unknown, and can change.

for example

[
    "FieldA", "FieldB", ...
]

image

Maybe there’s another way to perform this type of aggregation - but i’m currently stuck (maybe also just out of ideas for now) on how to perform this type of thing.

I need to build this to detect unique values in a dataset (unique based on a subset of columns).
currently, i need to build a seperate pipeline / pipeline segment for each table - allowing to specify this as parameter would greatly simplify our pipelines.

1 ACCEPTED SOLUTION

alchemiz
Contributor III

Hi Matthias,

Good day, what I did before was put the streaming documents in an array (you can use the Gate snap) then loop through the array doing the aggregate and grouping

See attached poc pipeline

input:
image

output:
image

test_2021_11_10.slp (13.4 KB)

Thanks,
EmEm

View solution in original post

2 REPLIES 2

alchemiz
Contributor III

Hi Matthias,

Good day, what I did before was put the streaming documents in an array (you can use the Gate snap) then loop through the array doing the aggregate and grouping

See attached poc pipeline

input:
image

output:
image

test_2021_11_10.slp (13.4 KB)

Thanks,
EmEm

ptaylor
Employee
Employee

Matthias,

I’m afraid the Aggregate snap doesn’t have the flexibility you’re looking for. While each field name in each row of both fields tables can be dynamically derived using an expression, you can’t dynamically populate the entire content of these tables.