I understand why you don’t want to change the scope of pipeline parameters, etc… And ALL ETL tools have little quirks that make it to where you can’t even really compare them to another product. In some ways you may always be better than your competitors even as you may be worse in others.
I have a little workaround though, and it is great. It is used all over the computer industry. How about a global variable scope? That would have allowed me to do what I originally wanted, and I am sure would have helped a lot of others out, etc… The problem with the current methods is that a pipeline is used independently from any need for the variable, so a given object will use it from the moment it starts. And the copy and join can get messy on more complex mappings for what amounts to be a local variable that generally must be passed.
If we had a GLOBAL variable that was like the pipeline parameters, but accessed only on an explicit reference, it would solve the problems many have and not add any new ones. You can even forbid its use for a pipeline parameter to avoid that concern. If one wanted to use it for a pipeline parameter, they could save it to the OTHER variable type, and use THAT. That variable would be local, and the pipeline parameter could use it, just as it does now.
One case where I could have used this in my current project was in saving a generated filename for an email. Using the copy/join could be a mess, and I ended up recreating it for the mail. It would have been nice to simply set a value, and use it in the mail, like I do in programs. There are other cases as well. Take that copy/join, for example. I often have to have a real key for it, but since I often am dealing with JSON return values, I have to parse the key out of that information, which luckily is designed into the json.