Forum Discussion

viji28's avatar
viji28
New Contributor
4 years ago
Solved

JSON data size fix

My source data has more than 1M record which is in a csv format. All those records had error , hence were routed to error view. Now , All those records needs to be logged in a S3 folder. Also i send...
  • ptaylor's avatar
    4 years ago

    I assume you are using the JSON Formatter to write the file, yes? But how are you formatting it? And what application are you using to try to open the file?

    By default, the JSON Formatter will use a very compressed format with no line breaks. Some editors don’t deal well with a file where all the data is on one very long line.

    You could enable “Pretty Print” on the formatter, which produces a much more readable and verbose format like this:

    But this format can also be a challenge for some JSON-capable applications since all of the data is inside a single JSON array.

    You might want to consider the “JSON Lines” format, where each line is a compactly formatted JSON object representing one document:

    This is often the best choice when dealing with a “log”. Each JSON line corresponds to a line of your CSV file. See https://jsonlines.org/