Certain flow snaps like “Filter”, “Head”, “Tail” when used in a complex pipeline doesn’t generate a preview while validating the entire pipeline however these generate a preview if the pipeline is simple.
For instance, if I’m trying to read data from an excel file with 100k+ records and use either ‘filter’ or ‘head/tail’, I can see the preview when the pipeline has been validated but it doesn’t generate one when the pipeline is huge. Any specific reason behind this?
I would want to see the preview as the pipeline that I’m currently developing is a complex one (multiple snaps prior to “flow” snaps), thereafter it needs a ‘filter + head/tail’ (i.e. flow snaps) and then it will again require a bunch of other snaps after those ‘filter + head/tail’ making it complex again.
Help on this matter would be highly appreciated.
Keep in mind that Preview only uses a subset of data (50 records by default), so the data being used my not meet the criteria being used.
An org admin can change the number of records used during preview in Manager > Settings. See https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438287/Data+Preview+from+a+Pipeline#DataPreviewfromaPipeline-SelectingtheNumberofRecordsforPreview
Agreed @dmiller on the 50 records by default on preview.
Preview has been working on all the snaps but for certain it doesn’t so I was curious to know why. There is always a workaround like exporting the data in an excel file right after those “flow snaps” but it is time consuming so even 50 records by default gives me an idea whether those flow snaps have worked correctly or not.
Head/trim might not require a preview however filter snap does need one; it works well if the pipeline is not complex but as soon as the pipeline grows, filter wouldn’t generate a preview which is kind of weird to me.
Sharing a snap below for your reference:
Even the successive snaps are not generating any preview which was happening before the flow snaps.
And here I do have a preview on Filter…
Sometimes I think you need to shift-click Validate to refresh the data.
Exactly, even I get the preview for flow snaps but as I mentioned earlier, works only when the pipeline is not complex.
What @dmiller probably meant, is, that the 50 records validating doesn’t meet the criteria in the filter.
Try increasing the preview count.
I’m pretty sure there’s nothing wrong with the Filter snap, and to prove that, you can try and add
true as a filter expression and re-validate. You should see all 50 records passed through.
Thank you @bojanvelevski for the suggestion. As far as I remember, the filter criteria has been kept in such a way that it should have passed all the records however lemme try your suggestion and keep you posted on this thread.
Regarding a simple pipeline, preview worked as seen below:
@dmiller: I tried with Shift + Validate but it didn’t work, I will go ahead with @bojanvelevski’s suggestion for now.
Please keep in mind that this expression will pass all of the records, and it’s just for debugging purposes.
Absolutely, thanks @bojanvelevski.
Putting the condition as true worked, thanks a lot for your help
Sharing the snap below for your references:
Also, you can always change the number of records the platform processes during validation:
Where did you put the condition as TRUE?
@amit.saroha: Condition = True was kept in the filter snap. If you’re following the thread, the snap after finalOutput is the filter snap (that’s where the condition was put).