Forum Discussion

ncrouch's avatar
ncrouch
Former Employee
9 years ago

Performing an Action when there is no data

A common integration pattern is to do something when no data is received. For example, we might read a file, parse it, and find that no records meet some filter criteria. As a result, we might send an email, or insert a ticket into a ticket management system like ServiceNOW.

However, in SnapLogic, this can be somewhat more difficult than it seems initially because of the streaming architecture. In fact, many snaps will not execute without input documents - rather hard to accomplish when there is no document to flow through:

So, how can we take an action, even though there’s no document to run the snap?

The trick is to make a document and force it into the stream with a Join Snap (set to Merge):



Note in this figure that even though nothing flows into the top view of the Join Snap, it still produces an output.

This enables us to use the Router snap to check whether or not a field that we know will exist in the source does in fact exist. If it does, we know that data has passed through the Filter snap and then been merged with the Sequence data. If it does not, we know that only the Sequence data has been passed through and therefore nothing made it through the Filter. Only one of these views will have anything pass through it.

The magic here is in the Merge option in the Join snap. As long as it is receiving data on a view, it will output - even if it has nothing to ‘join’ to. Meanwhile, the Sequence snap will always output the number of documents that you set it to. In this case, only one is required.

(This pipeline here: Do something after no doc filter_2017_04_26.slp (11.1 KB) )

sample_pipeline
transform

14 Replies

  • dmiller's avatar
    dmiller
    Former Employee
    9 years ago

    Thanks @ncrouch. I’ve been meaning to add this functionality to one of my pipelines and hadn’t got around to it yet. This saved me some time. 🙂

  • nganapathiraju's avatar
    nganapathiraju
    Former Employee
    9 years ago

    Why use the filter in first place? You can use the router to achieve the same functionality as the filter.

    Whatever continues from filter can be passed through one branch of the router. The other branch can handle the non-filtered data.

    I am assuming you dont want to take the same action on both filtered and unfiltered data. Your actions will be different based on the data whether it meets your filter criteria.

    • ncrouch's avatar
      ncrouch
      Former Employee
      9 years ago

      If you want to perform an action If and Only If no data is returned as a result of an action, you’ll need this.

      In essence, you’re saying “if nothing matches this filter, do (thing)”. It’s necessary because if you have a situation where no data is returned from a snap (whether a filter or a router or a REST Get or a SQL Select or …), the snaps following will not execute at all. This gives you the ability to handle that situation.

      • nganapathiraju's avatar
        nganapathiraju
        Former Employee
        9 years ago

        I see your point. This is definitely a nice way since our platform is a streaming platform.!!

  • murphy-brian's avatar
    murphy-brian
    New Contributor III
    8 years ago

    I tried the Sequence with a Joiner and have found the Joiner adds about 200-300ms to my pipeline, just to handle no results from a filter? Seems like a lot of overhead to overcome an issue from the filter pausing the pipeline flow. Naveen, I saw your comment on just using a router, but not sure how that would solve this if we are trying to create a single stream of data with less documents. I’d rather solve this in the source snap, but right now looking for an option that does not add a lot of overhead or pause the pipeline if the filter removes all documents. I was expecting the filter to be the bottleneck not the Joiner, if I use a union the performance is fine, but it won’t work because the first document is alway the output of sequence. Any ideas?

  • Ksivagurunathan's avatar
    Ksivagurunathan
    Contributor
    8 years ago

    We have started utilizing this tool in some of our ETL process and we are facing the exact same problem. If we are adding workaround like this, we may to need to add it more than once in each and every pipeline we develop. I sincerely hope Snaplogic take some quick action to handle situation like this across the platform. Adding workaround is not best option in this case. I wonder how others are handling this issue

    • tk42's avatar
      tk42
      New Contributor III
      8 years ago

      In the past we had used a Script Snap to detect/notify of the “nothing to do” condition. But it turns out that there are far less than 1% of our processes that the business required this information and, further, we recently discovered that no one that expressed the requirement is actually paying attention to the notifications.

      Short version: It’s not as big of an issue as people make it out to be.

      TK

  • sanam's avatar
    sanam
    New Contributor
    6 years ago

    Hi All,
    I have a similar requirement.
    I want to send email only if there is data output from the query and if there is no output I need to exit.
    I am not able to configure the router.
    I have select column_name from source where count=0
    if there is output sent mail
    below is the query
    Select tablename, record_count from audit where record_count=0.
    below is the screenshot of mapping and router config.

    • somdas's avatar
      somdas
      New Contributor
      6 years ago

      Seems like typo. Its hasOwnProperty()

  • Abhi's avatar
    Abhi
    New Contributor
    5 years ago

    " This enables us to use the Router snap to check whether or not a field that we know will exist in the source does in fact exist "

    Well what happen if we are not sure which field is coming out of source i.e field name is not fixed ?

    • cjhoward18's avatar
      cjhoward18
      Employee
      5 years ago

      You can use eval() to dynamically check for the existence of a property.

      This expression $.hasOwnProperty(eval('$dynamicKey')) checks for the existence of the key dynamicKey in the input document.

  • ash42's avatar
    ash42
    New Contributor III
    4 years ago

    hey I also tried using the

    so had a doubt will this cause it to execute this snap multiple time?

    • sdoscher's avatar
      sdoscher
      New Contributor III
      3 years ago

      @ash42 this will not cause the snap to execute multiple times. What you see there means that the snap before this is passing multiple documents TO the router. The router will “route” the multiple documents downstream according to the rules configured in the router.

  • ChristopheFr's avatar
    ChristopheFr
    New Contributor II
    2 years ago

    Hello,
    Thanks for the solution. I use it in some cases.

    But I have a pipe where the datasource are variables (queries on a DB). In that case, it could not work.
    Do you have any idea ?
    Thanks.