Forum Discussion
The in.TotalCound and out.totalCount functions will returns total number of documents that have passed through all of the Snap's.
About the resource utilization metrics, check the following post/pattern that uses the APIs for getting the current resource utilization.
Max - The simplest way that I can think to accomplish what you're asking for is to use a Gate snap with the default settings and rename the input view to "exchangeRates" as below:
A word of caution on the Gate snap with the default settings - this snap consumes all input into a single document, which may consume significant memory resources if processing a large volume of data.
Hope this helps!
koryknick- The gate snap worked in this use case. However, it is concerning that it may have resource issues if processing large data. The data set in this case is around 35K records. But we have a lot of cases this logic will need to be used and our datasets are huge. Is there a better way/more efficient way of doing this with big result sets?
It depends on the use case. You can use the Structure or Mapper snaps to manipulate individual input JSON documents, or use Group By Fields or Gate snaps to accumulate JSON documents into arrays.
SnapLogic is well suited to work with large data sets, you just need to think about how it is being processed and whether or not you need to accumulate everything into large, nested JSON documents for your endpoint.