cancel
Showing results for 
Search instead for 
Did you mean: 

Compare current to previous document in stream

kmiesse
Contributor

If I want to compare two consecutive documents in an ordered stream (compare the current to the previous or next), is the best way to do that with array indexing functions in mappers/conditionals?

5 REPLIES 5

tlikarish
Employee
Employee

What kind of comparison do you want to do? You’d probably need to do some kind of grouping on the documents upstream before you do the comparison.

kmiesse
Contributor

I’m assuming the documents have the same structure but not the same values. I just want to see the differences in the values between two consecutive documents in a single stream.

By default SnapLogic’s Snaps don’t have multiple documents available at the same time (there are exceptions, for instance the unique snap, which compares one document with the last, and if identical, passes only one out). If you want to do that, you’d have to use a script snap, and read multiple documents, or build your own snap which does the same with java.
You could always do some convoluted copying and joining streams with n+1 as the lookup value, kinda jerky, but could achieve the same, fairly efficiently if you were not processing large volumes of data.

Compare one row to the next, in streaming fashion with node code.

Take a look at this pipeline, it enables you compare an input document with the next, in a streaming fashion.

6cf6ef03ea94a2aef71fe883ee522c0bd7cdf188.png

First thing I’m doing is adding the index to the input stream in the first mapper, using the following expression:

4ff045169b28258de434d7bb702e1bbad11fcad9.png

In the second mapper in the top part of the flow, I’m getting the index of the target row I want to join it with (TargetKey): 5532a8942ad8606897235ee4b64a5eecb5da0390.png

So that we don’t have the first row in the right-hand side of the Join, I skip the first document:

And then I join based on the Index and TargetKey

3719ea18a148078306d8add5bc5be4961048095c.png

And in the output of the join, I see both documents in one, so I can compare the results:

38c8d0d17ae4306905efb489a7224076510307c8.png

CS new pipeline 1_2020_04_23.slp (8.8 KB)