Elastic Mapping with a simple Script Snap
An Elastic Mapper which outputs JSON that only contains keys that have non-null values
Background: The use case came while trying to map fields from a Salesforce Outbound Message. This message woul...
Forum Discussion
Hi Amar,
Sorry for the delay. After reading about the two things that you want to (parse data from the filename and parsing the data) I had a conversation with a colleague to verify some of my thoughts, and get more advice. We may need to create two pipelines “A” and “B” in pipeline A We switch from MultiFileReader to DirectoryBrowser. This is because Directory Browser will give non binary data. The resulting document will contain fields like name, type, size, path. You could hook this into a PipelineExecute Snap. Which will execute Pipe “B” You can pass this snap the “path” field.
In Pipeline “B” You can use a FileReader Snap to read the “path” field. Your file that gets returned, you said was a .txt file, although it is delimited with tabs and so should be able to be treated like a CSV file. So you should be able to hook up a CSV Formatter Snap to this File Reader. Then you should be able to process the data in this Pipeline “B”.
You could also ramp up the pool size, and let the Pipeline Execute Snap run multiple executions in parallel. But for starters, let’s see how it runs with one thread.
Thanks,
-Charlie
Ok. And you’ve set up the Mapper to map a string from the input document to the special field
content, like this?
Why do you need the Transcoder? The ASCII character set is extremely limited (only 128 characters) so if there are characters in the input that can’t be transcoded to ASCII that would be why you’re seeing that error. (The error message is confusing.)
