amar
New Contributor II
Joined 5 years ago
User Profile
User Widgets
Contributions
Hold Rest Post Snap until all records uploaded
Hi, Below is my pipeline where iam trying to upload data from oracle execute query which returns 200k records. iam grouping them into 10k at a time to Rest Data Upload snap(rest post). My question - is there a way i can stay on Rest Data Upload snap until all 200k records load. Tried using script snap to sleep for 10min but for some reason, script snap getting triggered after first 10k records and going to subsequent snaps.Re: Getting Snap aborted error when executing script file
Iam having similar issue when using script to just hold snap for sometime using below python script time.sleep: Import the interface required by the Script snap. from com.snaplogic.scripting.language import ScriptHook import time class TransformScript(ScriptHook): def init(self, input, output, error, log): self.input = input self.output = output self.error = error self.log = log # The "execute()" method is called once when the pipeline is started # and allowed to process its inputs or just send data to its outputs. def execute(self): self.log.info("Executing Transform script") while self.input.hasNext(): try: # Read the next input document, store it in a new dictionary, and write this as an output document. **time.sleep(1000)** inDoc = self.input.next() outDoc = { 'original' : inDoc } self.output.write(inDoc, outDoc) except Exception as e: errDoc = { 'error' : str(e) } self.log.error("Error in python script") self.error.write(errDoc) self.log.info("Script executed") # The "cleanup()" method is called after the snap has exited the execute() method def cleanup(self): self.log.info("Cleaning up") The Script Snap will look for a ScriptHook object in the “hook” variable. The snap will then call the hook’s “execute” method. hook = TransformScript(input, output, error, log) ERROR: Snap was aborted Hide Details… Script[5b4c5f504f619d3b40a380bf_85ddd35d-6fad-4333-88f9-e99472a24362 – e36724a0-b3b6-4523-8c0a-6f325c393b16] `com.snaplogic.cc.snap.common.ThreadDetails: prio=4 Id=38901 TIMED_WAITING at java.base@11.0.5/java.lang.Thread.sleep(Native Method) at org.python.modules.time.Time.sleep(Time.java:472) at jdk.internal.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) at java.base@11.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@11.0.5/java.lang.reflect.Method.invoke(Method.java:566) at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:190) at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:208) at org.python.core.PyObject.call(PyObject.java:461) … at java.base@11.0.5/java.lang.Thread.sleep(Native Method) at org.python.modules.time.Time.sleep(Time.java:472) at jdk.internal.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) at java.base@11.0.5/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@11.0.5/java.lang.reflect.Method.invoke(Method.java:566) at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:190) at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:208) at org.python.core.PyObject.call(PyObject.java:461) at org.python.core.PyObject.call(PyObject.java:465) at org.python.pycode.pyx172.execute$3(:31) at org.python.pycode.pyx172.call_function() at org.python.core.PyTableCode.call(PyTableCode.java:173) at org.python.core.PyBaseCode.call(PyBaseCode.java:306) at org.python.core.PyBaseCode.call(PyBaseCode.java:197) at org.python.core.PyFunction.call(PyFunction.java:485) at org.python.core.PyMethod.instancemethod___call(PyMethod.java:237) at org.python.core.PyMethod.call(PyMethod.java:228) at org.python.core.PyMethod.call(PyMethod.java:218) at org.python.core.PyMethod.call(PyMethod.java:213) at org.python.core.PyObject._jcallexc(PyObject.java:3565) at org.python.core.PyObject._jcall(PyObject.java:3598) at org.python.proxies.builtin$TransformScript$172.execute(Unknown Source) at com.snaplogic.snaps.script.CommonExecute.execute(CommonExecute.java:123) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:770) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:552) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:836) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:402) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:116) at java.base@11.0.5/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.5/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.5/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.5/java.lang.Thread.run(Thread.java:834) Error Fingerprint[0] = efp:java.lang.4yCCeq43` Script[5b4c5f504f619d3b40a380bf_85ddd35d-6fad-4333-88f9-e99472a24362 – e36724a0-b3b6-4523-8c0a-6f325c393b16] `com.snaplogic.cc.snap.common.ThreadDetails: prio=4 Id=38901 RUNNABLE at org.python.core.PyBaseCode.getline(PyBaseCode.java:319) at org.python.core.PyTraceback.(PyTraceback.java:29) at org.python.core.PyException.tracebackHere(PyException.java:171) at org.python.core.PyException.tracebackHere(PyException.java:158) at org.python.core.Py.setException(Py.java:1619) at org.python.pycode._pyx172.execute$3(:31) at org.python.pycode._pyx172.call_function() at org.python.core.PyTableCode.call(PyTableCode.java:173) … at org.python.core.PyBaseCode.getline(PyBaseCode.java:319) at org.python.core.PyTraceback.(PyTraceback.java:29) at org.python.core.PyException.tracebackHere(PyException.java:171) at org.python.core.PyException.tracebackHere(PyException.java:158) at org.python.core.Py.setException(Py.java:1619) at org.python.pycode.pyx172.execute$3(:31) at org.python.pycode.pyx172.call_function() at org.python.core.PyTableCode.call(PyTableCode.java:173) at org.python.core.PyBaseCode.call(PyBaseCode.java:306) at org.python.core.PyBaseCode.call(PyBaseCode.java:197) at org.python.core.PyFunction.call(PyFunction.java:485) at org.python.core.PyMethod.instancemethod___call(PyMethod.java:237) at org.python.core.PyMethod.call(PyMethod.java:228) at org.python.core.PyMethod.call(PyMethod.java:218) at org.python.core.PyMethod.call(PyMethod.java:213) at org.python.core.PyObject._jcallexc(PyObject.java:3565) at org.python.core.PyObject._jcall(PyObject.java:3598) at org.python.proxies.builtin$TransformScript$172.execute(Unknown Source) at com.snaplogic.snaps.script.CommonExecute.execute(CommonExecute.java:123) at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:770) at com.snaplogic.cc.snap.common.SnapRunnableImpl.execute(SnapRunnableImpl.java:552) at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:836) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:402) at com.snaplogic.cc.snap.common.SnapRunnableImpl.call(SnapRunnableImpl.java:116) at java.base@11.0.5/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.5/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.5/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.5/java.lang.Thread.run(Thread.java:834) Error Fingerprint[0] = efp:org.python.core.WHDYXyoj`3KViews0likes0CommentsRe: Date Validation using Expression Library
I didn’t see any solution on it. Can someone please help out on validating date parameter to certain format. In my example, i have parameter $start_date where i need validate whether it matches ‘MM/DD/YYYY’ or not. Iam looking some kind of javascript expression that routes to error if format fails. Thanks1.4KViews0likes1Comment
Groups
Higher Ed Users Group
This user group is for customers within higher education.0 Posts
