Replace your two scripts with mappers should net you some performance gain. You don’t need to use a script to make a string of json.
Script in “Split events” pipeline which is doing
data = self.input.next()
newData = {}
newData[‘record’] = data
newData[‘sequence_id’] = data[‘sequence’][‘id’]
newData[‘event_id’] = data[‘event_id’]
self.output.write(newData)
replace with a mapper like
Script in write events is like
data = self.input.next()
self.output.write(data[‘record’])
replace with a mapper like
Thanks Dwhite
I’ll make that change, but I checked the average execution time for both the scripts , it’s very less (0.141 sec).
Also, how should we decide what to set in pool size property of Pipeline Execute snap.
Thanks
Hi @MikeP,
After the CSV parser, are your headers with empty values like "" ? Or simply there’s no data after after the csv parser ?
hi j.evegelvski, the files will always have header text values like “Emp_ID”, the rows after this will be empty. I cant identify if we have any data for this file situation - no data. My dev head is saying maybe a ‘row count’ property (if data in any rows) after header row filtered out, this would be handy, but doesn’t seem to exist. Then just check if this value > 0.
Addendum: After the CSV Parser snap, then the next snap is a straight mapper snap for the document ($) the output view is
Hence why I thought the expressions I mentioned initially would work.
Hi @MikeP,
In that case your expression should work because $.isEmpty() should return true. Not sure why it doesn’t work for you, but you can also try with the following: $.values().length == 0. Can you also share your settings in the router snap ?
