We query DB (collects the events generated by Parent system) and grouping the records based on field “Type”. Using GroupN, I get all the records for that particular type. Using this list of recordValues, we retrieve rest of the information from Parent system. This design is working well as groupN is helping us to retrieve all the records in a single query for all the recordValues retrieved in that instance of SQL.
However, there is one scenario where for a particular type of event, there is additional information that comes along in the event.
When we do GroupN for this event type, grouping is happening at individual fields and thus the correlation between recordValue and additional information is getting lost.
The work around we did is to process individual query to the parent system just for this event type. But this approach is degrading the performance as rest of the snaps are waiting for this event type to complete.
[
{
“record_id”: 53538,
“event_type”: “GROCERIES”,
“objectInfo”: “Milk”
},
{
“record_id”: 53539,
“event_type”: “SPICES”,
“objectInfo”: “Paprica”,
“noLongerCarry”: “Salt”
}
]
GroupN works when the object doesnt have “noLongerCarry” as I can query all the information based on objectInfo and create the output document which carries multiple recordID information in one single document.
But when noLongerCarry object gets generated in any of the event_type, groupN is losing the correlation. This is forcing us to process such event individually and thus making multiple trips to the source system and generating a document for each entry.
Is there a mapping technique that solve this issue?
