Slow PostgreSQL Update Snap
I have a PostgreSQL Update snap at the end of a pipeline. It’s job is to update a column in a row if nothing in an inbound row (matching that data) has been modified. This means that most of the time...
Forum Discussion
kumar25 - Do you just need something like the following?
input-1 line1
input-2 line1
input-3 line1
input-1 line2
...etc?
I've attached an example pipeline how to accomplish this. Basically, it's adding variables to track the input the data is coming from and the record number of each record. Once the data is combined using Union, sort the data on the record number and input view. Finally, remove the temporary values used for this process. Note the use of the "Passthrough" option in the Mappers so I don't need to know what the record layout is.
Hope this helps!
PS - I appreciate Aleksandar_A 's contribution; however, I do warn against using the Gate snap with the "All input documents" setting. If your input dataset is large, it can consume considerable resources on your execution nodes, causing other pipelines to pause to wait on resources, or worst case can crash the node depending on other activity. Gate is a powerful snap and can be used very effectively, just remember:
You should be able to reference the other library through the
libvariable. Is that not working?For example, if you this as the library
helper.expr:{ chefify: x => x + ' bork! bork! bork!' }You should be able to reference it from another library, like so:
{ msg: () => lib.helper.chefify('Hello, World!') }Note that if the references are not in functions, but directly in the top-level expression, you’ll need to ensure the libraries are listed in the right order in the pipeline properties.
I found a better way to achieve what i was look for.
But thanks for highlighting this.@tstack
I have a use case where a child pipeline has multiple expr files.
I want to pass a value to the child pipeline using which my chiled pipeline would choose one of the expression library.
is there a way to do it ?In the child pipeline properties, add a parameter that you’ll use to specify the library to load. Then, in the “Expression Libraries” section, add a path that uses the parameter you just created. You’ll probably also want to set the name of the imported library using the “As” column. In the screenshot below, the parameter is
configand defaults to the library filedev.exprand the library can be referenced vialib.confin expressions:In the parent pipeline, you’ll pass the
configparameter with the library paths that you want. In the screenshot below, there are two documents entering the PipeExec with “dev.expr” and “prod.expr” as the paths. So, the child will execute first with the “dev” library and then with the “prod” library.
Is there a way to declare methods with global scope in expression library.
Is nesting multiple libraries together a good preactice ?Do you mean without the
lib.libnameprefix? No, libraries are always imported under thelibobject.I’m not sure what you mean by this, can you give an example?
