Need your help /solution to solve the problem.
When we trying to add more lookup’s in the pipeline performance is drastically decreasing . To load some 30K records with 3 lookup snap, it took around 30 minutes. Instead of lookup, if i use the join instead of lookup pipeline has completed within 3 minutes.
source --> Redshift Lkp1 (Good performance) --> Redshift Lkp2 (Slow) --> Redshift Lkp3 (Very slow) --> Target
Instead of Redshift , I tried with SQL server lookup and getting the same issue… Also, interchanged the lookup’s and tried it… That’s also not working…
Do you have any idea, what could be the reason for this performance issue?.