Parameters and other values should be usable as constants are

For example, on HEAD, or TAIL, the number of documents, AND the offset should be able to use a value passed in or a parameter. The value of the snap is severely curtailed by not having that functionality. I have been using them to easily do a test on full data and figured I would tie them into my parameter system and was shocked when it provided an error.

ALSO, the number of documents should be settable to zero for NO documents. I came up with a nice idea to provide full or only applicable reports, and run all data, nothing, or just a subset, and this makes that harder to do and not worth it at this point.

there are probably other cases where this could be used. It may even be good to add it for retries and timeouts as that could maybe be tuned.

Also, it would be nice if the zip and similar snaps could stop on a record, rather than a byte, so a test(preview) would run with no error on the partial data. It took me a while to figure out what was going on, and it would be nice to be able to always use the same settings.

Yes, these properties should be expression-enabled. In the meantime, you can substitute a Filter for the Head snap and check the snap.in.totalCount property. As an example, to get the first ten documents, you would use the following “Filter expression”:

snap.in.totalCount <= 10

Or, with a parameter:

snap.in.totalCount <= parseInt(_limit)

That won’t work well for the Tail snap since it maintains state. You’d probably have to resort to a GroupByN snap with a size of zero to collect all the docs into a single one and then slice it from there.

Can you elaborate? Are you talking about the ZipFile Reader snap? During validation, a binary snap is allowed to write a fair amount of data downstream so that a parser snap can process the input without getting a short read. The binary snap doesn’t really understand the structure of the stream, so it’s difficult for it to figure out where the end of a record is.

I am talking about things including the zipfile reader snap. Actually, it doesn’t even BOTHER to find where a record ends. If it were merely a short read, that would likely be ok, but it errors out. It acts like that is the end of the file, and there is corruption, when it simply stopped reading because there is a limit in preview mode.

Because of this, if I have a problem with a record, even if it is the first one, I must parse it out, create a new TEST dataset, and point things to that, and hope the issue is with that record. I didn’t know about that filter expression variable. OK, I will try that.