CMeyers
New Contributor II
Joined 9 years ago
User Profile
User Widgets
Contributions
Re: Accessing Elastic Search through Snaplogic
Elasticsearch is entirely REST based, so anything you need to do can be done via the REST snaps. Also if you have a secured elasticsearch cluster with Shield, which uses basic auth, then a basic auth REST account allows you to login.1.2KViews0likes0CommentsRe: Pipeline execution in series
If you mean you want all docs to go through the first before running the second, you could do a tail snap in between the pipeline executes to grab the last document out of the first before running the second. If you need the document output, so basically wanting to do batch as opposed to SnapLogic’s typical way of streaming data through, you could instead put a sort snap with unsorted input which will force the snap to read all documents first before sorting. Note that means all the documents will remain on the server for a time while they are gathered to sort, so would need to be sure you have the server resources to do so.1.1KViews0likes1CommentRe: Incoming request xml is getting transformed into json
Documents in snaplogic are always in JSON when coming out of a circle connector (as opposed to the diamond connectors). In this case the XML string that is generated is put on an xml field in the generated json document, so if you want to send that to a REST call you would have to properly reference that field.4.6KViews1like0CommentsRe: REST GET Snap error (SLDB does not support data larger than 100 MB)
We had the same issue. This occurs because the preview data is written to SLDB and has a 100mb limit, which is usually fine for the 50 rows of data, but for large REST responses that contain an internal array of items within a single document that limit is easily reached. This is however only a preview time error, as a full run does not create previews, so actually running the pipeline worked fine and did not cause the error.2.3KViews1like0CommentsRe: JSON Object Diff
I have code that can do this in javascript, although for arrays it is checking differences by index so you would need to be aware of that. I would think this code snippet could be adapted to run in a script snap and get the output you desire. I put in the two objects from your example into the little utility code I had and it found the differences, would just need to modify the output formatting to what you specified. Function - self.internalFindDifference = function(obj1, obj2, parent, accum) { var result = accum || { "changes": [] }; for (key in obj1) { if (!obj2.hasOwnProperty(key)) { result.changes.push({ "field": parent ? parent + "." + key : key, "type": "add", "value": obj1[key] }); } else if (!_.isEqual(obj1[key],obj2[key]) && typeof obj1[key] == 'object' && typeof obj2[key] == 'object') { var newParent = parent ? parent + "." + key : key; arguments.callee(obj1[key], obj2[key], newParent, result); } else if (!_.isEqual(obj1[key], obj2[key])) { result.changes.push({ "field": parent ? parent + "." + key : key, "type": "change", "fromValue": obj2[key], "toValue": obj1[key] }); } } for (key in obj2) { if (!obj1.hasOwnProperty(key)) { result.changes.push({ "field": parent ? parent + "." + key : key, "type": "delete", "fromValue": obj2[key] }); } } return result; } Example calling code - var result = self.internalFindDifference(obj1, obj2, null, null); Console.log(JSON.stringify(result, null, 4));1.4KViews0likes5Comments