Join/Union missing data

I have 2 single field documents containing string representations of time (Ex: “12:03:43”) that when I use a Join snap with any join type, or when I use a Union, I see that there are items that don’t make it in the join.

02%20PM 19%20PM

I have tried various things and no matter how I join these two documents containing ‘time’ strings there are data-points dropped in the join.

I also have seen simple copy snaps drop data as well. I don’t have screenshots of this but it seems that running it through a ‘Unique’ snap resolved that issue, although I don’t know why that should matter.

Have you verified this is also the case when you execute the pipeline? One thing you might be hitting is the preview document limit. This is specified in the designer settings. The default is set to 50. Is your setting larger than the amount of data being input into the Join?

What could be happening is that the Join is reading the first N documents on the left (upper) input. If 15:54:38 is after N, then it won’t appear in the preview output.

This must be the issue.
It looks like when I execute this doesn’t happen, but I was reluctant to do so when it appeared that data was being dropped.

1 Like