How much Load can SnapLogic handle?
I understand that SnapLogic is not perceived as a typical Migration tool for one time migration (i.e. an ETL tool) but more as an integration tool (iPaaS) for Integrations. Is that true ? If not, can SnapLogic be used for One time Migration and how much load can it handle ? For e.g. say we want to bring in a few million records in SnapLogic say Customer Master for the sake of this conversation, apply business rules and transformations on source data fields and then send it to downstream systems. Will SnapLogic be able to handle such a high volume load with transformations ? In short, can it be used for like an ETL tool that is typically used for One time Data loads as well? Thanks in advance.
Empty custom metrics from Prometheus adapter, AWS EKS
Following the documentation here https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/2017558845/Deploying+a+Groundplex+in+Kubernetes+with+Elastic+Scaling No resources are returned from the custom metrics and the snaplex helm chart cannot be installed. { “kind”:“APIResourceList”, “apiVersion”:“v1”, “groupVersion”:“custom.metrics.k8s.io/v1beta1”, “resources”: } Does anyone have an existing working setup with some of the latest versions of Kubernetes/Prometheus? Is it possible do scale using other tools (Keda, Datadog etc.) and is there documentation for it? Any feedback is much appreciated.
Recycle Bin
I’m reading that items are only stored in the Recycle Bin for 30 days in the SL online documentation. I see items much older than 30 days in our org. Is there a setting for this somewhere I’m not seeing? Why would I be seeing deleted items older than 30 days in the Recycle Bin? Thanks! Chris
How to access error view document
Normally with our pipelines, if there is an error, we feed the error view document into an email, or a Jira object We have had a third party create a pipeline recently, and they have not done this We have had an error over night, and this is telling me to open the error view document, and I found a post online that suggests you can do this from the pipeline, but doesn’t give any details, and I can’t find anything in Snap Logics help, or anything else online Is this possible? And how do I do it? Thanks
GHES | SnapLogic | Additional features after installation
We are interested in understanding the potential benefits of subscribing to GHES and installing it. Specifically, we would like to know if GHES offers any additional features beyond the existing OOTB Git pull integration provided by SnapLogic. If there are indeed additional features, we would appreciate it if you could provide us with a comprehensive list and explanation of those features.
MS-SQL Server Stored Procedures
I have a pipeline that creates a staging table in a SQL database that is then compared to a target table using a SQL server stored procedure. It is important that the staging table be completely populated before the stored procedure is executed, and there is a gate after the staging table insert and before the stored procedure call. Five percent of the time the pipeline works as expected. All other times the staging table insert hangs when about half the records are loaded. The pipeline does not report an error but the jcc error log has a line (see below) that seems to be relevant. {“ts”:“2023-05-16T14:50:39.197Z”,“lvl”:“ERROR”,“fi”:“SnapRunnableImpl.java:1034”,“msg”:“Snap execution failed unexpectedly”,“exc”:“com.snaplogic.api.ExecutionException: Batch operation failed\r\n\tat com.snaplogic.snap.api.ErrorViewsImpl.write(ErrorViewsImpl.java:264)\r\n\tat com.snaplogic.snaps.sql.SimpleSqlSnap.process(SimpleSqlSnap.java:496)\r\n\tat com.snaplogic.snap.api.ExecutionUtil.process(ExecutionUtil.java:106)\r\n\tat com.snaplogic.snap.api.ExecutionUtil.executeForDocument(ExecutionUtil.java:118)\r\n\tat com.snaplogic.snap.api.ExecutionUtil.execute(ExecutionUtil.java:81)\r\n\tat com.snaplogic.snap.api.SimpleSnap.execute(SimpleSnap.java:70)\r\n\tat com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:812)\r\n\tat com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:586)\r\n\tat com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:877)\r\n\tat com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:436)\r\n\tat com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:120)\r\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\r\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:834)\r\nCaused by: com.snaplogic.snap.api.SnapDataException: Batch operation failed\r\n\tat com.snaplogic.snaps.sql.SimpleSqlSnap.process(SimpleSqlSnap.java:493)\r\n\t… 15 more\r\nCaused by: com.snaplogic.api.ExecutionException: Batch operation failed\r\n\tat com.snaplogic.snap.api.ErrorViewsImpl.write(ErrorViewsImpl.java:264)\r\n\tat com.snaplogic.snap.api.sql.DatabaseUtils.writeErrorView(DatabaseUtils.java:933)\r\n\tat com.snaplogic.snap.api.sql.operations.JdbcOperationsImpl.insert(JdbcOperationsImpl.java:1009)\r\n\tat com.snaplogic.snaps.sql.SimpleSqlInsertSnap.executeInsert(SimpleSqlInsertSnap.java:209)\r\n\tat com.snaplogic.snaps.sqlserver.Insert.processData(Insert.java:91)\r\n\tat com.snaplogic.snaps.sql.SimpleSqlWriteSnap.processDocument(SimpleSqlWriteSnap.java:202)\r\n\tat com.snaplogic.snaps.sql.SimpleSqlSnap.process(SimpleSqlSnap.java:484)\r\n\t… 15 more\r\nCaused by: com.snaplogic.snap.api.SnapDataException: Batch operation failed\r\n\tat com.snaplogic.snap.api.sql.DatabaseUtils.writeErrorView(DatabaseUtils.java:926)\r\n\t… 20 more\r\nCaused by: net.jodah.failsafe.FailsafeException: java.lang.InterruptedException: sleep interrupted\r\n\tat net.jodah.failsafe.SyncFailsafe.call(SyncFailsafe.java:165)\r\n\tat net.jodah.failsafe.SyncFailsafe.get(SyncFailsafe.java:69)\r\n\tat com.snaplogic.snap.api.sql.operations.JdbcOperationsImpl.flushBatch(JdbcOperationsImpl.java:1943)\r\n\tat com.snaplogic.snap.api.sql.operations.JdbcOperationsImpl.insert(JdbcOperationsImpl.java:1005)\r\n\t… 19 more\r\nCaused by: java.lang.InterruptedException: sleep interrupted\r\n\tat java.base/java.lang.Thread.sleep(Native Method)\r\n\tat net.jodah.failsafe.SyncFailsafe.call(SyncFailsafe.java:162)\r\n\t… 22 more\r\n”,“snlb”:“Encrypted+cdiwork±+insert”,“snrd”:“fd47ae71-0f96-468d-b307-aec4d2b336c0”,“plrd”:“5dba84c8017c0e2416c17c86_85c2f064-2c91-46b7-bf78-769511a1bd5a”,“prc”:“Encrypted cdiwork - insert[5dba84c8017c0e2416c17c86_85c2f064-2c91-46b7-bf78-769511a1bd5a – fd47ae71-0f96-468d-b307-aec4d2b336c0]”,“xid”:“cf1027ce-b460-de45-0974-1f51a2a727f4”}