Hi I’m attempting to connect to a SalesForce instance using a SOQL snap, read all the table names, then use a pipeline execution snap to trigger a child pipeline that dynamically looks up all the field names for each table name it is fed, then pulls all the data for that table with another SOQL snap and pipes it to AWS S3. I have got parts of it working but sometimes the queries are too long and error out with SalesForce. Does anyone have a template or example of how to do this besides manually setting up a pipeline for every table?
Hi Travis - check out this pattern in our patterns catalog. Hope this helps : Ingest data from Salesforce application into AWS Cloud Storage (S3)