Writing a forEach
Hello,
I have a use case where I am getting my input in tab delimited form, which after doing a groupBy N I am converting to an array so that I can make some comparison operation.
I need to extra...
Forum Discussion
Thanks krupalibshah for sharing the sample data. I would like to know how big is your input file ? Reason I am asking you this because in Snaplogic I am not sure if it is easily doable because in your case when you are making an array you need to keep track of last record too so that you know in which array you have to put the record. I was thinking if a script task can do this for you but it will not because script task works on single record. However, if your input file is not that big you can easily do this using any programming/scripting language like Python.
Actually, a script task doesn’t really work on a single record. It is simply that the example has you read in one record at a time.
For example, I had to create a special CSV script to replace the CSV script that snaplogic comes with. That script reads in, and keeps, the first row as a header, and reads the others using values from the first row. That first row could easily have been a thousand, and joining with the first row could have been processing all the data in the snap.
The size limit is based on your system, and various things like how soon you want it. Unfortunately, I think a lot of memory is not freed until you finish, though I could be wrong. Of course you should try to limit data requirements.
Basically, the snap has five processing points.
- Before you read any data.
- As you read the data.
- At some breakpoint you determine.
- Errors/exceptions as they occur.
- At the end.
And you can write out the records to any area as you want on your conditions and timing.
Steve
