Hi Team,
I have a requirement to pass the document from parent pipeline to child pipeline, in child pipeline it has only one snap(Mapper Snap).
Document which will be passing from parent pipeline should go as a input to mapper snap in child pipeline and provide the mapper output to parent pipeline back which will be using in parent pipeline for further processing of data.
Can someone help on the above requirement.
@Nagaraju.Julakanti Pipeline execute will take care. You have to use mapper before Pipeline Execute in parent pipeline and in your child pipeline can use mapper/ other snap and use $filedname in expression text box to get the value from parent pipeline.
It’s working thanks.
Wait a second. @Nagaraju.Julakanti, can you please post images of what you’re doing?
To have a child pipeline containing nothing but a single mapper snap generally makes no sense to do, unless you have a special reason to force the mapping to run on a different node or snaplex – a scenario that I cannot imagine being useful.
If it’s a single snap, why not just put the single snap inline in the parent pipeline where the interface to the child snap currently is?
What am I missing here?
One Scenario could be - If your mapper has Heavy Transformations then you can move it to child pipeline and have pool size set to greater than 1 to improve the performance.
Thanks.
-Su.
Thanks, It will help us as we have too many columns and transformations.
@ [ForbinCSD],
We have a requirement to write data from one file to other file based on the mapping provided in the csv file as the number of columns in data file is huge we are generating a mapper snap dynamically as a separate pipeline using the approach suggested in below topic(Using Templates to drive Smartlink or JSON output) and calling the child pipeline in parent pipeline to get the mapper output.
Please suggest if any other better approach to generate mapper dynamically using input csv file.
Ah. Now it’s becoming a bit more clear what you’re doing and why. Thanks!
Nope, I don’t know right offhand of a better way to handle that.
One Scenario could be - If your mapper has Heavy Transformations then you can move it to child pipeline and have pool size set to greater than 1 to improve the performance.
Unlikely. In fact, it might make it slower, due to the overhead of launching child pipelines and distributing the load across them.
Using Pipeline Execute will definitely be slower than putting the Mapper directly in your pipeline, due to the overhead of Pipeline Execute. Hopefully, the performance with Pipeline Execute will be adequate. But make sure you check “Reuse executions to process documents” on the Pipeline Execute to get the best performance. This will launch just one child pipeline (or N child pipelines if you set Pool Size = N) and feed all of the input documents to that instances (or those N instances). Otherwise, it will create a new pipeline instance for every single document, which is FAR slower.
@ptaylor - Sorry I missed the Re-use Checkbox. We did see quite a performance improvement with a pipeline on setting greater pool size and re-use checkbox checked.
That’s good. Yes, I didn’t mean as a general rule. It very much depends on what the child pipeline is doing. If it’s just a Mapper, I still think it’s unlikely you’d see much improvement. In my test, increasing the Pool Size increased the time.