Designing and Running Pipelines

Efficient pipeline design for reading a large amount of data from an on-premises RDBMS (JDBC) and write it as multiple files in S3

We have a use case where we need to read a large amount of data from an on-premises RDBMS (JDBC) and write it as multiple files in S3. No matter what, the data has to travel across the network from our on-premises data center to AWS. I’m looking for ...

Conditionally Remove or Add a Field

Hello, How do you conditionally remove (or add) a field to a document? E.g., I have a mapper: Expression → Target Path 20 → $field1 10 → $field2 ‘’ → $field3 null → $field4 So I now have: [ { ‘field1’:20 ,‘field2’:10 ,‘field3’:‘’ ,‘field4’:null } ] N...

Resolved! Compare two files to see if they are the same

Does anyone have any experience or recommendations about how to compare two files and check if they are identical? I don’t mean data files, I’m thinking about media files here so something along the lines of creating a hash of the files and comparin...

Resolved! SOAP Execute has no output

Following the documentation and tutorial (Youtube): https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1437963/SOAP+Execute I am not able to view the output (small green box at the right of SOAP Execute) of the SOAP call. I followed all of the...

Error pipeline STOPPED - WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer

Hi SL gurus, I designed an async pipeline, see picture. I created an error pipeline and defined it in my async pipeline properties. I created a triggered task for my async pipeline. Now, to simulate an error, I put some trash into JSON Formatter. Whe...

"Sort path" based on dynamic columns // sort & diff

Hi, i’m having the following challenge: I’m pulling data from a REST API as well as SQL tables. Before grabbing the actual objects, i’m grabbing “Object Types” from the REST API to know which tables to grab from the SQL DB. The tables each have diffe...

"Pipeline execution failed or was aborted" - proper design of async interface(?)

Hi SL gurus, I designed an async pipeline, see picture. I created an error pipeline and defined it in my async pipeline properties. I created a triggered task for my async pipeline. Now, to simulate an error, I put some trash into JSON Formatter. Whe...

Rest Get Pagination Issue

Hello, I am having issues with the Rest GET Pagination mechanism. The Pagination works fine in the validate mode, but doesn’t traverse after 1st page when i actually run the pipeline. This is so weird that i am able to see the pagination working fine...

SnapLogic - Integration Nation

Forum Posts

Efficient pipeline design for reading a large amount of data from an on-premises RDBMS (JDBC) and write it as multiple files in S3

Conditionally Remove or Add a Field

Resolved! Compare two files to see if they are the same

Resolved! SOAP Execute has no output

Error pipeline STOPPED - WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer

"Sort path" based on dynamic columns // sort & diff

"Pipeline execution failed or was aborted" - proper design of async interface(?)

Rest Get Pagination Issue

Data Validator Pattern error

Deleting multiple files at once from SFTP folder

SQS -pipeline stalling

Best Approach for Handling CSV Import with Truncat...

Fetch dashboard/monitor details via a pipeline

mapValues returns null when adding certain conditi...

What account should be used for API with query par...

How to dynamically update an account?

Execute each child pipeline multiple times

Routing files from S3