Forum Discussion
Simple answer: no.
In theory, snaps like the CSV Parser could be enhanced with a new setting to limit the number of output documents, but there’s little reason to burden snaps with this additional complexity when you can achieve the same result by adding a Head snap.
May I ask why it’s preferable to stop reading the file early?
In my pipeline I’m performing some analysis on those first N records and writing the analysis results to a file. The input file is very large and the output file containing the analysis results doesn’t appear to be closed until the pipeline completes (when the file reader finishes reading the file). The amount of time between the analysis of the first N records being complete and the file reader finishing reading the file is significant; at least significant enough that our user has complained about it.
- PSAmmirata5 years agoEmployee
@ptaylor - Once this change is implemented, is the Exit snap the best way to programmatically stop the pipeline once the Head’s output view is closed?
Why do you need the pipeline to stop if the output file containing the analysis results has been written and closed?
But, yes, I suppose you could add an Exit snap after the File Writer if you do want the pipeline to exit. But even if you don’t, the File Writer will have completed and written the file that your user needs, even while the Head continues to consume its input and the pipeline continues to run.
- PSAmmirata5 years agoEmployee
If we no longer need the pipeline running, I don’t want to use Snaplex node resources unnecessarily. Also, users question why is the pipeline still running once it produced the desired file.
Related Content
- 3 years ago
- 4 years ago
- 6 months ago