Hi Syed M. and Marwen M., Thank you both for taking the time to review the issue and for sharing your suggestions. I wanted to provide an update on my findings. Initially, I re-ran the pipeline, but the same error persisted. To further investigate, I executed the same pipeline in the lower environment, and it completed successfully without any issues. However, when I pointed the production source accounts to the lower environment and attempted to load the data into QA, the pipeline failed again with the same "Error writing input to the data file" and "Broken pipe" error. This made it difficult to identify the actual root cause, as the error message did not clearly indicate the underlying issue. To troubleshoot further, I routed the failed records to the error view and reviewed the detailed error information. From there, I discovered that the actual failure was caused by an Oracle index being in an UNUSABLE state, which resulted in the following database error: ORA-26026: unique index <index_name> initially in unusable state After identifying the index issue, the root cause became clear. My question for the community is: Is there a more efficient approach to identifying the underlying database error in Oracle Bulk Load Snap failures? In this case, the top-level error reported was "Broken pipe," which was misleading and required additional troubleshooting in a lower environment to uncover the actual Oracle error. In a future high-priority production incident, we may not have the flexibility to perform this kind of investigation. Are there any recommended debugging techniques, Snap settings, logging options, or best practices that can help expose the underlying Oracle error more quickly when using the Oracle Bulk Load Snap? Thank you again for your support and guidance.
even after rerun same error
Hi Everyone, Could you please help me understand and resolve the error below? The Oracle Bulk Load Snap is failing with the error mentioned below. I am unable to identify the root cause or understand what is triggering the issue. Any guidance or suggestions would be greatly appreciated, as I need to resolve this as soon as possible. Thank you in advance for your support and assistance. Error writing input to the data file Resolution: Please file a defect against the snap Reason: Broken pipe Hide Details... Write Oracle STG2 Price[5d8e00959248c560fbcf7117_b13e2019-844b-4471-8e60-be1659dd06fa -- 5c10b3fb-0778-496c-b961-eecbbc1a5a37] com.snaplogic.snap.api.SnapDataException: Error writing input to the data file at com.snaplogic.snaps.oracle.BulkLoad.processDocument(BulkLoad.java:628) at com.snaplogic.snaps.sql.SimpleSqlSnap.process(SimpleSqlSnap.java:494) at com.snaplogic.snaps.oracle.BulkLoad.execute(BulkLoad.java:528) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:836) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:599) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:901) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:447) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:119) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Broken pipe at java.base/java.io.FileOutputStream.writeBytes(Native Method) at java.base/java.io.FileOutputStream.write(Unknown Source) at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) at java.base/sun.nio.cs.StreamEncoder.implFlush(Unknown Source) at java.base/sun.nio.cs.StreamEncoder.flush(Unknown Source) at java.base/java.io.OutputStreamWriter.flush(Unknown Source) at java.base/java.io.BufferedWriter.flush(Unknown Source) at com.snaplogic.snaps.oracle.BulkLoad.processDocument(BulkLoad.java:626) ... 13 more Reason: Broken pipe Resolution: Please file a defect against the snap Error Fingerprint[0] = efp:com.snaplogic.snaps.oracle.ctb9jSj5 Error Fingerprint[1] = efp:java.io.UaHETxDF Write Oracle STG2 Price[5d8e00959248c560fbcf7117_b13e2019-844b-4471-8e60-be1659dd06fa -- 5c10b3fb-0778-496c-b961-eecbbc1a5a37] com.snaplogic.cc.snap.common.ThreadDetails: prio=4 Id=346649 RUNNABLE at java.base@11.0.24/java.lang.StringBuilder.append(Unknown Source) at java.base@11.0.24/sun.net.util.URLUtil.urlNoFragString(Unknown Source) at java.base@11.0.24/java.security.CodeSource.<init>(Unknown Source) at java.base@11.0.24/java.net.URLClassLoader.defineClass(Unknown Source) at java.base@11.0.24/java.net.URLClassLoader$1.run(Unknown Source) at java.base@11.0.24/java.net.URLClassLoader$1.run(Unknown Source) at java.base@11.0.24/java.security.AccessController.doPrivileged(Native Method) at java.base@11.0.24/java.net.URLClassLoader.findClass(Unknown Source) ... at java.base@11.0.24/java.lang.StringBuilder.append(Unknown Source) at java.base@11.0.24/sun.net.util.URLUtil.urlNoFragString(Unknown Source) at java.base@11.0.24/java.security.CodeSource.<init>(Unknown Source) at java.base@11.0.24/java.net.URLClassLoader.defineClass(Unknown Source) at java.base@11.0.24/java.net.URLClassLoader$1.run(Unknown Source) at java.base@11.0.24/java.net.URLClassLoader$1.run(Unknown Source) at java.base@11.0.24/java.security.AccessController.doPrivileged(Native Method) at java.base@11.0.24/java.net.URLClassLoader.findClass(Unknown Source) at org.eclipse.jetty.webapp.WebAppClassLoader.foundClass(WebAppClassLoader.java:642) at org.eclipse.jetty.webapp.WebAppClassLoader.loadAsResource(WebAppClassLoader.java:615) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:529) at java.base@11.0.24/java.lang.ClassLoader.loadClass(Unknown Source) at com.snaplogic.snaps.sql.SimpleSqlSnap.handleSnapException(SimpleSqlSnap.java:525) at com.snaplogic.snaps.sql.SimpleSqlSnap.process(SimpleSqlSnap.java:502) at com.snaplogic.snaps.oracle.BulkLoad.execute(BulkLoad.java:528) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:836) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:599) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:901) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:447) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:119) at java.base@11.0.24/java.util.concurrent.FutureTask.run(Unknown Source) at java.base@11.0.24/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base@11.0.24/java.util.concurrent.FutureTask.run(Unknown Source) at java.base@11.0.24/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base@11.0.24/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base@11.0.24/java.lang.Thread.run(Unknown Source) Error Fingerprint[0] = efp:java.lang.DuhpYimn
Hi Lee D. Thank you for the details — that really helps! Yes, I would definitely like to see the implementation you’ve created. It would give me a much clearer picture of how to structure the solution, especially how you combined the SnapLogic List Snaps with the public runtime API. Please share the implementation whenever you can. Thanks again for offering to help! Cheers, Niranjan
Hi Team, Currently, we manually check the Pipeline Executions in SnapLogic Monitor to track the last run date, completion time, and status of each pipeline. We then prepare an Excel report and send it manually through email. I want to automate this entire process so that we receive an email containing the last execution date, time, and status of all pipelines without doing these checks manually. My question: 👉 What is the best way to automatically gather the latest execution details of all pipelines from PROD and email a consolidated report? If possible, please provide a POC or example approach to achieve this. Thanks!
Thanks Vaidehi Lad and Slavko Zdravevski
Hi, I am working in migration project from Datastage to snaplogic. In DataStage, the Remove Duplicates stage lets me specify one or more key columns to determine uniqueness (with an upstream Sort on the same keys). It then keeps the first/last occurrence per key depending on sort order. In SnapLogic, I tried using Deduplicate / Unique (and related approaches), but I’m not getting the same record counts or the same “kept” record per key. Notably:
The Unique snap (in my setup) seems to treat the entire document when determining duplicates, so if non-key columns differ, it doesn’t drop them—unlike DataStage, which only looks at the key.
Deduplicate snap doesn’t give me deterministic control over which record is kept when duplicates exist unless I pre-sort and shape the data.
Minimal Examples What DataStage does (Remove Duplicates on key columns) Input (CSV/table):
id,name,city,updated_at
1,Alice,NY,2024-01-01
1,Alice,NY,2024-03-15
2,Bob,SF,2024-02-10
2,Bob,SF,2024-02-05
3,Carol,LA,2024-01-20Goal (keep newest per id): Sort upstream by id ASC, updated_at DESC
Remove Duplicates on key = id
Output:
id,name,city,updated_at
1,Alice,NY,2024-03-15
2,Bob,SF,2024-02-10
3,Carol,LA,2024-01-20DataStage ignores non-key differences when deciding duplicates and keeps the first row per key after sort (hence deterministic). What I’m seeing in SnapLogic (Deduplicate/Unique) Input (as documents): JSON [ {"id":1,"name":"Alice","city":"NY","updated_at":"2024-01-01"}, {"id":1,"name":"Alice","city":"NY","updated_at":"2024-03-15"}, {"id":2,"name":"Bob","city":"SF","updated_at":"2024-02-10"}, {"id":2,"name":"Bob","city":"SF","updated_at":"2024-02-05"}, {"id":3,"name":"Carol","city":"LA","updated_at":"2024-01-20"} ] Observed issues: If Unique considers the entire document, both id=1 rows are treated as distinct because updated_at differs → wrong count vs. DataStage.
Using Deduplicate alone (without shaping) doesn’t guarantee keeping the newest row per id.
Kindly provide me resolution to remove duplicate same as datastage.
Thanks! Ingo
I have attached screenshot where it has total 12 fields and 3 rows of data I need first 9 fields headers to appear which is in highlighted in yellow and fields headers of last 3 fields which is highlighted in green should not appear in file. but the data should appear as it has data in last row.
Hi James Thanks for the suggestions — really appreciate your inputs! We’ll check with IBM to confirm whether this limitation is coming from DB2 itself or the driver, as you mentioned. Regarding the approach of streaming records one by one: would this significantly impact performance, especially when handling larger datasets? Just want to be sure we’re considering the trade-offs before implementing that strategy. Thanks again for the guidance!