Scheduled Task + File Poller guidelines

[hrender]

I have several legacy file-based interfaces I’m migrating to Snaplogic. Many of them involve polling for new input files on a scheduled basis, usually something like every 30/60/120 seconds. I’ve coded the ones I’ve done using a combination of a scheduled task that calls a pipeline containing a file poller. I’ve played around with having all of the iteration done by the task and having the file poller only run once, and I’ve also tried setting the task to run every 30/60/90 minutes and having the file poller run every 30/60/120 seconds with a timeout equal to the task frequency. I like doing the latter since I can iterate at a higher frequency, but given that task scheduling is somewhat approximate, getting the the two schedules in sync is more of a problem.

Is there any general guidance as to how to set up the two schedules, i.e. is it better to schedule a task to run once an hour/day/week and then have the poller do all of the smaller iterations in that larger time-slice? If so, what should the task frequency generally be?

Thanks in advance.

[marenas]

@koryknick thank you for the response. I too am not an expert and I really appreciate all your thoughts on this.

The above settings turned out to be prone to data errors in my case. The directory has over 52k records. When I enable Only Output On Change, the pipeline kicks in every time there are changes in the contents of the directory even though the changes do not relate to the file that I am looking for, and then polls continually (due to pulling timeout -1). It eventually finds the matching file however the file is not the most updated one. I am trying this configuration below and I will play around them depending on the results. btw, the scheduled task is set to run every 5 mins for this.

I will take note of your recommendation to move files to another location for processing.

[koryknick]

Do you have the File Filter in the snap configured with a specific filename? You should not get results unrelated to the file you specify.

[marenas]

Yes, I have a file filter. I am re-enabling this "Only Output On Change’, set the polling timeout to 0, and exit on the first match property. I set a scheduled task yesterday that runs the pipeline every 5 minutes. So far I don’t see any unexpected results, but I will keep on monitoring this.

thank you for the follow up @koryknick