I’d like to request that expressions be allowed in the ‘number of documents’ and ‘document offset’ settings of the Head and Tail snaps. Here’s my use case.
I’d like to create a parameterized, reusable pipeline that reads a file and breaks it up into smaller pieces. The name of the file being read and the size of the pieces (number of rows) are pipeline parameters. The pipeline would include a mapper snap that computes the name of the file piece, a router snap that would route the documents for a given file piece to a specific file writer, and each file writer snap would be preceeded by a head snap with the number of documents dynamically configured, so that when number of documents of a given piece have been reached in the head snap, the output view of the head snap will be closed, and the downstream file writer snap will complete. This eliminates the need for the file writer snap to continue running, even though all of the records have been written to it, while the other piece files are being written. I’m attaching a pipeline that helps illustrate my use case.file_split_2021_04_21.slp (383.2 KB)