Flatten JSON files into CSV files
Created by @schelluri The pipeline pattern flattens a JSON file, which has multiple objects, and turns it into a CSV file. Configuration Sources: JSON Generator Targets: CSV file Snaps used: JSON Generator, JSON Formatter, JSON Parser, Script, CSV Formatter, File Writer Downloads MS_Flatten_Script.slp (31.4 KB)
Remote file renaming/moving
Hi, I have a requirement to push a file to SFTP server. But since the file is quite large and to avoid race condition (we writing and remote function reading at the same time) we like to write the file with different extension or to a temp folder in remote server and them rename or move the file at the server. None for the File snaps support these feature. Could you please help me identify a way around to address above problem.
JavaScript Function Scoping (Script Snap)
I thought about creating a ticket to ask this, then realized others probably could use this information as well… When writing a script for a JavaScript snap, one can define all the subroutines as member functions within the “impl” object: var impl = { input: input, output: output, error: error, log: log, // A real function would do a lot more than this doSomething: function( inputVal ) { return inputVal; }, execute: function() { while (this.input.hasNext()) { var doc = this.input.next(); ... var foo = this.doSomething(doc.baz); ... var wrapper = new java.util.HashMap(); wrapper.put("original", doc); this.output.write(doc, wrapper); } } }; However, some developers prefer making functions stand by themselves when they do not access other members of the object (thus, not needing to be a member themselves): // A real function would do a lot more than this function doSomething( inputVal ) { return inputVal; }; var impl = { input: input, output: output, error: error, log: log, execute: function() { while (this.input.hasNext()) { var doc = this.input.next(); ... var foo = doSomething(doc.baz); ... var wrapper = new java.util.HashMap(); wrapper.put("original", doc); this.output.write(doc, wrapper); } } }; My Question: In terms of scoping (and memory usage), is there any difference between the two? Particularly if there are multiple script snaps running at the same time in the same pipeline? For example, suppose you have two script snaps, Script-1 and Script-2: Clearly, var impl is a separate instance in each, otherwise you couldn’t have two script snaps in the same pipeline. If each snap has it’s own separate function doSomething() in the separate scripts, I’m assuming those are also outside of each other’s scope, is that correct? Since functions are first-class objects in JavaScript, does it consume more memory to make the functions external to the impl object, than to have them as members? Inquiring minds want to know!Powershell command execute Snap
Hi, Is there any way to execute a command line scripts (Ex: Power shell)? Our scenario is to execute a set of commands (which is located over in a server, where we could establish a connection through an FTP/SFTP) at the end of load completion for an UI Refresh, please advise if there is any direct way or just let us know any workaround!! Thanks, AravindConfiguring the Script Snap to use a configured HTTP proxy environment variable
HTTP-compatible Snap Packs can leverage an HTTP proxy configured in the Snaplex’s Network Proxies configuration tab within the SnapLogic Manager web application. However, the Script Snap is different because you can write Scripts to call external processes (e.g. curl) and these will not be aware of any proxy configuration set within the SnapLogic application. curl can be configured to use a proxy directly via the --proxy argument, but if you wished to enforce that proxy usage across all usages of the Script Snap, you can set the http_proxy and/or https_proxy environment variables within a special file - /etc/sysconfig/jcc . Environment variables declared within this file will be visible to the Snaplex application (OS-level env vars will not be). This file (and directory) may not exist in your Snaplex, so you may have to create them (similar to the instructions on the Configuring a custom JRE version page): sudo mkdir -p /etc/sysconfig; sudo sh -c "echo 'export http_proxy=username:password@proxy-ip-address:port' >> /etc/sysconfig/jcc" substituting the equivalent values for username/password (if authentication is required), proxy-ip-address , and port (you may also want to add https_proxy too). Once this file is created, restart the Snaplex application ( /opt/snaplogic/bin/jcc.sh restart or c:\opt\snaplogic\bin\jcc.bat restart ) and the http_proxy / https_proxy environment variable will now be active within the SnapLogic product. Assuming your proxy is correctly configured, you can then run your Script that calls the external process and, if the process supports using a proxy, it will respect the setting. For example, the following Script Snap (Python) uses the subprocess library to execute curl and adds the response body to the output document. # Import the interface required by the Script snap. from com.snaplogic.scripting.language import ScriptHook import subprocess class TransformScript(ScriptHook): def __init__(self, input, output, error, log): self.input = input self.output = output self.error = error self.log = log # The "execute()" method is called once when the pipeline is started # and allowed to process its inputs or just send data to its outputs. def execute(self): self.log.info("Executing Transform script") while self.input.hasNext(): try: # Read the next input document, store it in a new dictionary, and write this as an output document. inDoc = self.input.next() proc = subprocess.Popen(['curl','https://www.snaplogic.com'], stdout=subprocess.PIPE) (out, err) = proc.communicate() outDoc = { 'original' : out } self.output.write(inDoc, outDoc) except Exception as e: errDoc = { 'error' : str(e) } self.log.error("Error in python script") self.error.write(errDoc) self.log.info("Script executed") # The "cleanup()" method is called after the snap has exited the execute() method def cleanup(self): self.log.info("Cleaning up") # The Script Snap will look for a ScriptHook object in the "hook" # variable. The snap will then call the hook's "execute" method. hook = TransformScript(input, output, error, log) On execution, the proxy access log should show the request being routed through the proxy.
Script help
Hello All, There is log file writting during execution of SnapLogic pipeline, as we dont have direct access so wanted to browse below directory from script snap. Can anyone help me here. Below is the javascript used for writ the logs:- try { load(“nashorn:mozilla_compat.js”); } catch (e) { } var impl = { input: input, output: output, error: error, log: log, execute: function () { while (this.input.hasNext()) { var doc = this.input.next(); this.output.write(doc, doc.response); var lastSystemTime = doc.lastSystemTime; var wrapper = new java.util.HashMap(); wrapper.putAll(doc); wrapper.remove("lastSystemTime"); var externalSystem = wrapper.externalSystem; if ((externalSystem != null) && (externalSystem.length > 0)) { if (! externalSystem[externalSystem.length - 1].endTime) { externalSystem[externalSystem.length - 1].endTime = lastSystemTime; } } wrapper.put("success", false); try { var om = new ObjectMapper(); om.writeValue(new java.io.File("/tmp/API_LOG/" + doc.executionId), wrapper); } catch (e) {} } } }; importPackage(com.snaplogic.scripting.language); importPackage(java.util.HashMap); importPackage(“com.fasterxml.jackson.databind”); importPackage(java.io.File); var hook = new com.snaplogic.scripting.language.ScriptHook(impl); Can any onne help me here to read above directory or files ?? Thanks in advanced.
