Showing results for 
Search instead for 
Did you mean: 

Calculate bytes written or read by a snap?

New Contributor II

There’s a way to calculate the total number of input or output documents, which is monumentally handy.

However, is there a way to calculate the number of bytes read on input/output? This would be particularly useful for file chunking ie. if you want to segment files written to S3 by 100mb files, as opposed to having to make a guess as to the number of documents you need to read.

I can imagine a way to do this right now roughly by building a Script snap to calculate every documents’ flattened JSON space requirements, but I worry this would actually slow down the pipeline a lot since it requires basically either traversing a document dynamically or by re-serializing each document in order to produce a byte count.

Is there another way we can get the number of bytes read or written on a snap?