My source data has more than 1M record which is in a csv format. All those records had error , hence were routed to error view. Now , All those records needs to be logged in a S3 folder. Also i send...
I assume you are using the JSON Formatter to write the file, yes? But how are you formatting it? And what application are you using to try to open the file?
By default, the JSON Formatter will use a very compressed format with no line breaks. Some editors don’t deal well with a file where all the data is on one very long line.
You could enable “Pretty Print” on the formatter, which produces a much more readable and verbose format like this:
But this format can also be a challenge for some JSON-capable applications since all of the data is inside a single JSON array.
You might want to consider the “JSON Lines” format, where each line is a compactly formatted JSON object representing one document:
This is often the best choice when dealing with a “log”. Each JSON line corresponds to a line of your CSV file. See https://jsonlines.org/
Coyote - this is actually a fairly tricky question. Since the SnapLogic expression language doesn't allow for multi-line syntax, the only solution without reverting to a Script snap would be to create an expression that can call itself recursively. This is where expression libraries will come in to help us. I apologize that I don't have a lot of time to completely detail out my answer, but will give you what I can in just a few minutes.
First, the solution. Please find the sample pipeline and expression library file attached here. Just download it, then upload the expr lib file to your project, then import the SLP with the example pipeline. You'll see your requested object as well as an additional array to test nested arrays containing objects. Basically I just wanted to test a more complex example to ensure I had that working correctly.
The call to the expr lib function is very simple:
I'm just putting the results back to the root object in the target path ($).
But let's take a look in the lib expr file itself to see how it's doing this:
lowerCaseKeys : (obj) =>
// Recursively convert all object keys to lowercase
obj
.mapValues((val,key) =>
match typeof val {
'object' => this.lowerCaseKeys (val)
, 'array' => val.map(elem => typeof elem == 'object' ? this.lowerCaseKeys(elem) : elem)
, _ => val
}
)
.mapKeys((val,key) => key.toLowerCase())
I'm first calling the Object.mapValues() method to check for any sub-elements that are objects or array of objects - in either case, I'm going to recursively call the lowerCaseKeys() function to do that next lower level of the nested object, leaving all the current values as-is.
After we're done with the mapValues() call, then I can do the rest of the keys at the root level using the Object.mapKeys() method.
I also included an upperCaseKeys() function in the expr file for completeness. There are other functions in the expr file that would take much longer to explain, but are just as useful as what I've explained here.
