04-11-2022 10:58 AM
Certain flow snaps like “Filter”, “Head”, “Tail” when used in a complex pipeline doesn’t generate a preview while validating the entire pipeline however these generate a preview if the pipeline is simple.
For instance, if I’m trying to read data from an excel file with 100k+ records and use either ‘filter’ or ‘head/tail’, I can see the preview when the pipeline has been validated but it doesn’t generate one when the pipeline is huge. Any specific reason behind this?
I would want to see the preview as the pipeline that I’m currently developing is a complex one (multiple snaps prior to “flow” snaps), thereafter it needs a ‘filter + head/tail’ (i.e. flow snaps) and then it will again require a bunch of other snaps after those ‘filter + head/tail’ making it complex again.
Help on this matter would be highly appreciated.
Regards,
Darsh
Solved! Go to Solution.
04-11-2022 11:22 AM
What @dmiller probably meant, is, that the 50 records validating doesn’t meet the criteria in the filter.
Try increasing the preview count.
I’m pretty sure there’s nothing wrong with the Filter snap, and to prove that, you can try and add true
as a filter expression and re-validate. You should see all 50 records passed through.
2 weeks ago
There are a couple of things that needs to be taken into consideration here:
Firstly, Snap Execution should be set to "Validate & Execute", any other settings wouldn't generate a preview for you.
Secondly, change the Preview Document Count to Max i.e. 2000 as seen below:
Thirdly, if you're dealing with a huge dataset let's say 50k and you only have 100 records out of 50k that satisfies the condition in the filter snap, you may or may not see them in the preview as the order of records displayed in preview is not guaranteed, you can use a SORT snap before filter if you know the first 50 will suffice the condition; if that's the case then you will see the output preview for sure.
Thank you.
Best Regards,
Darsh
04-11-2022 11:04 AM
Keep in mind that Preview only uses a subset of data (50 records by default), so the data being used my not meet the criteria being used.
An org admin can change the number of records used during preview in Manager > Settings. See https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438287/Data+Preview+from+a+Pipeline#DataP...
04-11-2022 11:08 AM
Agreed @dmiller on the 50 records by default on preview.
Preview has been working on all the snaps but for certain it doesn’t so I was curious to know why. There is always a workaround like exporting the data in an excel file right after those “flow snaps” but it is time consuming so even 50 records by default gives me an idea whether those flow snaps have worked correctly or not.
Head/trim might not require a preview however filter snap does need one; it works well if the pipeline is not complex but as soon as the pipeline grows, filter wouldn’t generate a preview which is kind of weird to me.
04-11-2022 11:11 AM
Sharing a snap below for your reference:
Even the successive snaps are not generating any preview which was happening before the flow snaps.
04-11-2022 11:15 AM
And here I do have a preview on Filter…
Sometimes I think you need to shift-click Validate to refresh the data.