Forum Discussion

travis_pendleto's avatar
travis_pendleto
New Contributor II
6 years ago
Solved

Ingesting SalesForce Data

Hi I’m attempting to connect to a SalesForce instance using a SOQL snap, read all the table names, then use a pipeline execution snap to trigger a child pipeline that dynamically looks up all the field names for each table name it is fed, then pulls all the data for that table with another SOQL snap and pipes it to AWS S3. I have got parts of it working but sometimes the queries are too long and error out with SalesForce. Does anyone have a template or example of how to do this besides manually setting up a pipeline for every table?