Output preview not available for Inner Join

@darshthakkar - as @ptaylor mentioned, when you are working with Pipeline Preview data and need to see data movement through the entire pipeline, you need to reduce your source data sets to a limited enough set that any snaps that can potentially filter data (i.e. Join, Filter, etc.) will still have data to pass. In your case, I would suggest putting a temporary filtering condition (i.e. Where Clause) on your database Select snap so that your data will match after the Join snap. Once you have verified that the pipeline logic is correct, simply remove the filter from your select and execute the entire pipeline and validate the complete results.

@darshthakkar - I think you may be running into a common misunderstanding with the Data Preview functionality. By default, SnapLogic pulls only the first 50 records for preview into each snap. If the two sources you are reading from do not have any records that align to meet your join criteria with an inner join, then nothing will be output.

For example, if SourceA contains records such as CustomerId from 1 to 50, and SourceB contains records with CustomerId from 2000 - 2050, then the inner join doesn’t match anything and all records are effectively filtered.

You can try updating your User Settings / Preview Document Count as high as 2000 to see if that helps:

Or you can set filter conditions on your source data to ensure alignment of data that will match in your Join - in the case of the CustomId values I gave, you could set a filter condition in your database select to pull only records with CustomerId values of 1-50.

To prove if my theory is correct, you can switch your Join to a Left Outer join type and see that documents will flow through to the output view of the Join.

That is correct - if the Mapper has no input, it can’t determine the schema.

If the numbers in the one input view start at 1000 and the numbers in the other input view are all just three digits, I would expect no output from the Join.

Seems like using mapper after Inner Join is practically impossible. Is there a workaround?

I haven’t gotten a chance to test this out but it’s on my to-do list so should be completed before the end of this week and I can keep you posted @koryknick on how it goes.

Thank you for your help on this one.

Regards,
Darsh

Output previews are very helpful while developing the pipeline as we can literally see the ouput of what we are developing, the workaround for this would be exporting the results in a flat file and analyzing the flat file thereafter, isn’t this time consuming and prone to errors (as there would be human intervention)?

If you were thinking of any other ways, I would request you to please share.

@koryknick: What if the inputs that go into an Inner Join have been sorted, would the ouput preview be randomized or would it be based on the sorted input?

I sorted both the inputs and I was expecting some values in preview but I didn’t. If output preview is purely randomized then we can disregard this scenario. Some snaps below for your reference:

Pipeline showing no output preview on Inner Join:

Snowflake Sort snap exported results for references:

Flat file Sort snap exported results for references:

Preview count is 50, if it would have been 1, I could understand nothing coming on an output preview:

Screenshot of Join settings:

Without the output preview, its challenging to go ahead with other transformations using mapper, sort, etc. What would you suggest using instead of this?

CC: @ptaylor

@ptaylor: What would you suggest to use instead of an Inner join then? I would need the output as there is a lot of transformations that I would need to do after the output of Inner join.

I believe, this might have been the case with Distinct results while saving/validating pipeline vs executing pipeline - #10 by ptaylor too but getting Null values was very surprising. Once I get a chance to test that, I will keep you posted on that thread itself. For now, let’s just stay focused on the workaround for Inner Join (it’s very easy for me to get distracted when the 2 phenomenon are somewhat similar)