ContributionsMost RecentMost LikesSolutionsHow to get all the projectspace list under a workspace that a user has access to This pipeline will give the list of all projectspace details under a workspace that the user has access to. the flow is like that- it will scan all project space details under a workspace. it will find out the unique list of group name those are involved in that workspace. 3.it will find out the member list of those group list. 4.it will filter the group list where the specified user is present. it will find the the project space where those filtered group list has access to. it will filter the project space list where the specified user has access to. 7.finally it will merge the result derived for group list and individual user permission. the configuration of some snap is below: Get list of all project space that one user has access to, under the organization Here is the pipeline design to get list of all workspace list that a user has access to. user permission can be given in 3 ways individual user permission grant group permission admin permission this pipeline will scan all workspace of that organization and it will also scan all group list of that organization and it will find out where are the user has access to. as this scanning takes lots of time we need to cache the data somewhere. here we are using file storage in snaplogic for this cache operation. the file will be refreshed after each 1 hr interval. we are using a scheduled task for this refreshing.the scheduled job pipeline is given at last. here are some configuration details of the main pipeline. this will try to read the data from local file storage first. if local file does not exist it will go to get all group list: then it will get group details for each group name. at next step it will save the extracted data in a file for future use and will proceed for further processing. here is the configuration of filter to get the workspace list where that mentiond user has access to. parallelly for individual user permission then the two result will be joined together to get the final result. the scheduled task pipeline is:it will be scheduled at cron of 1 hour interval. Create a SnapLogic task through another trigger task Here is the pipeline to create a trigger task for a existing pipeline and if we expose the pipeline as a trigger task then , through that trigger task we can create other trigger task for other pipeline. the input json is; [ { “parameters”: { “pipeline_path”: “/xxx/xxx1/fff/jsjs”, “pipeline_snode_id”: “GPLEX”, “job_type”:“regular”, “execution_timeout”:5, “include_external_urls”:true, “max_failures”:10, “max_in_flight”:200 }, “job_name”: “myjob1111”, “path_id”: “xxx/yyy/zzz”, “job_class”: “PipelineJob”, “note”: “test job”, “external_run_url”:“” } ] the configuration of the “snaplogic read(snap name:“alreday exist?”)” like- the configuration of snaplogic create snap is like: and configuration of get job url details is: This will create a trigger task for the given pipeline or if already exist it will throw error. Parquet Reader & Writer through azure S3 For saving data in parquet format in s3 , below is the pipeline configuration. This pipeline creates meta data from the data itself , though it uses parquet data type ‘binary’ which is equivalent to string. the first mapper converts the doc into string. we used the below arrow function into the 1st mapper: $.mapValues((value, key) => value==null?“”:value.toString()) the second mapper function creates the meta from the data. The arrow function used for this: $.keys().map(x=>{“col_name”:x,“data_type”:“binary”}) For reading the file back below is the configuration of parquet reader: Please note “Use old data format” may be critical otherwise it may fail to read. This is to be checked when data are not nested. Re: How to get meta data from a document set using script snap? The error in the script was due to partial iteration of the, input doc. The input doc is to be fully iterated otherwise this error will come at runtime like the input is not fully consumed. if required we can break the script into two script. How to get meta data from a document set using script snap? I want to get the meta data of a document using a script snap.i prepared the below javascript using script snap. Its working fine at validation time but failing at runtime. please find my script below. // Ensure compatibility with both JDK 7 and 8 JSR-223 Script Engines try { load("nashorn:mozilla_compat.js"); } catch(e) { } // Import the interface required by the Script snap. importPackage(com.snaplogic.scripting.language); // Import the Java utility classes. importPackage(java.util); /** * Create an object that implements the methods defined by the "ScriptHook" * interface. We'll be passing this object to the constructor for the * ScriptHook interface. */ var impl = { /* * These variables (input, output, error, log) are defined by the * ExecuteScript snap when evaluating this script. */ input : input, output : output, error : error, log : log, /** * The "execute()" method is called once when the pipeline is started * and allowed to process its inputs or just send data to its outputs. * * Exceptions are automatically caught and sent to the error view. */ execute : function () { this.log.info("Executing Transform Script"); // var i=0; if (this.input.hasNext() ) { try{ // Read the next document, wrap it in a map and write out the wrapper var doc = this.input.next(); var keyArray = doc.keySet().toArray(); var wrapperOut ={}; for( var index in keyArray) { var new_data = {}; var key = keyArray[index]; var type=(!isNaN(parseInt(doc.get(key))) ? 'int' : 'binary'); // wrapperOut.put("data_type",type); // wrapperOut.put("col_name",key); wrapperOut.data_type=type; wrapperOut.col_name=key; output.write(wrapperOut); // wrapperOut.add(new_data); } // wrapper.put("original", doc); // this.log.info("Transform Script finished"); } catch(err) { var wrapper = new java.util.HashMap(); wrapper.put("errorMsg", err); this.log.error(err); this.error.write(wrapper); } // i++; } // output.write(wrapperOut); this.log.info(" data"+wrapperOut.toString()); // this.output.write(doc, wrapperOut); } }; /** * The Script Snap will look for a ScriptHook object in the "hook" * variable. The snap will then call the hook's "execute" method. */ var hook = new com.snaplogic.scripting.language.ScriptHook(impl); Where is wrong with this script?Its failing at runtime giving below error: java.io.IOException: Failed to close subscriber If the document view is: items color price car red 1100 mango green 5 Then my desired output is: column name type items string color string price int