amardeep2021
New Contributor III
Joined 4 years ago
User Profile
User Widgets
Contributions
How to remove columns with null value alone in pipeline?
Hi, I have a scenario where I have few columns have null value few not. Objective is to not to bring in columns with null value for each records. source: id a b c d e abc null 1 2 null null def 11 null 22 33 null ijk 111 222 null 333 444 lmo null null null null 5555 Expected: abc 1 2 samething with other records as well. Does anyone know any trick on this? Please let me know. Regards, Amar.6.4KViews0likes6CommentsRe: How it do insertion faster for redshift
Hi Rajesh, I have batch and pool size in pipeline execute as 1. Looping total record count for each cycle is 70k. Any tips on what should be batch size and pool size to make this pipeline execution complete sooner? Please let me know. Regards, Amar.2.6KViews0likes0CommentsRe: How it do insertion faster for redshift
Hi Rajesh, Yes it’s Bulk load redshift. It checks for a condition in redshift select snaps and if it’s true, it self execute again till the condition is false. So the set of snaps in this pipeline will be executed again and again in loop till false. Other than updating account settings batch and pool size, is there a way to address this slow execution.? Regards, Amar.2.6KViews0likes0CommentsHow it do insertion faster for redshift
I am trying insert 70K records in loop till a condition is false. Looping logic working fine. But, I see it takes 1 hour to insert 70K records each time. PFA of the pipeline I am executing. Is there a way to perform insertion faster? Any tips would be helpful. Regards, Amar.4.5KViews0likes10Comments