Thank you @bojanvelevski, should it be > or ‘<’ ?
Now when I reviewed your original post, yes, it should be “<”. You want to consider dates that are less than 10 hours than the current date.
That’s correct and I can explain you in detail.
Let’s assume, I manually execute this pipeline, the current date and time is June 29, 10:40am EST, with the filter condition, we should only consider records whose Last_updated was before 48hrs than the current date and time.
Let’s assume again that 2 records have Last_Updated as June 27, 6pm EST and the difference is 48hrs (it’s easy to explain if we are considering values (i.e. 24, 48, 72) that are multiple of 24) and the other records have Last_Updated as June 25, 10:40am EST, we should ONLY get those 2 records as those fall into 48hr bracket.
Than I go back to my original statement. This expression will do the job:
Date.parse($Last_Updated) > Date.now().minusHours(48)
I wasn’t able to fetch any records with this criteria but that doesn’t mean this is not the solution. It is indeed and thank you @bojanvelevski for your help on this one.
Reason behind no records to be fetched was there were actually no records those were updated in the last 48 hours, I tweaked my criteria to 7 Days and found some for testing purposes and I can confirm that this worked.
Also, with Date.parse, the date format is consistent so not to worry about EST and UTC.
Something new I learned today was
Date.now().minusDays(7)
This above function helped me test the functionality for the last 7 days.
Thanks again @bojanvelevski , signing off from this thread now!
Moreover, the Timezone of Last_Updated is UTC as I haven’t changed it to any other timezone, may be that is creating a hurdle?
Another approach would be to put the parts of the pipeline you wish to parallelize in a child pipeline that is called with the Pipeline Execute snap. Using a child pipeline will save you the trouble of having to maintain copies of the snaps and make it easier to test different concurrency levels since it’s just a matter of changing the ‘Pool Size’ property.
So is this the same as having a sub-pipeline and tune the pool size of the Pipeline execute?
This is in addition to this.
So if you have 100 documents, and only one pipeline execute with a pool size of 10, then 10 are distributed for each instantiation of execution.
In this case net instantiations are only 10.
Now with the same amount of documents, you have 2 routes each calling pipeline execute with a pool size of 10 then 50 are distributed in 2 routes and then 5 run across each instantiation of execution.
In this case net instantiations are now 20.
Hope that makes sense.
So then why I have such duplicate in terms of snap and why not just do one pipeline execute with pool size of 20?
