How to use 3rd party python libraries in python Script

Question

Please can some shed some light on using a third party library (boto3, requests) in a python script
Thank you in advance!

bhavin · Answer

AFAIK 3rd party py libs like boto3 are not currently supported within py script, neverthless you can still invoke your your py scripts that has ref to boto3 libs, the flow is
1 - create py scripts using your fav editor, outside snaplogic and save it on all of snaplogic nodes (aka jcc)
2 - get UnixExec snap from you SnapLogic account team (this is a field snap that you can request from SnapLogic), this will help you run py scripts out side of SnapLogic via ssh on JCC node/system
3 - Add UnixExec snap in your pipeline, where ever you need to invoke your py script

jaybodra · Answer

Thank you so much Bhavin!
I am assuming I can pass pipeline variables as parameters to my python script like we run it on shell. Please can you shed some light on this.

bhavin · Answer

Please help me understand your usecase, what are you trying to do?

jaybodra · Answer

this is something I am trying to do,
processing url using python lib and writing results to s3. I need to pass url, sub_dir_name, doc_name and delimiters as argument to the script

bhavin · Answer

AFAIK you can call 3rd party java libs inside script snap using py as scripting language, here is a sample script that uses aws java sdk, gets a list of s3 objects from a bucket
# Script begin
# Import the interface required by the Script snap.
import java.util
import json
import sys 
sys.path.append('/opt/snaplogic/userlibs/aws-java-sdk-1.9.6.jar')
from com.amazonaws import AmazonClientException
from com.amazonaws import AmazonServiceException
from com.amazonaws.regions import Region
from com.amazonaws.regions import Regions
from com.amazonaws.services.s3 import AmazonS3
from com.amazonaws.services.s3 import AmazonS3Client
from com.amazonaws.services.s3.model import Bucket
from com.amazonaws.services.s3.model import GetObjectRequest
from com.amazonaws.services.s3.model import ListObjectsRequest
from com.amazonaws.services.s3.model import ObjectListing
from com.amazonaws.services.s3.model import PutObjectRequest
from com.amazonaws.services.s3.model import S3Object
from com.amazonaws.services.s3.model import S3ObjectSummary
from com.snaplogic.scripting.language import ScriptHook

class TransformScript(ScriptHook):
    def __init__(self, input, output, error, log):
        self.input = input
        self.output = output
        self.error = error
        self.log = log

# The "execute()" method is called once when the pipeline is started
# and allowed to process its inputs or just send data to its outputs.
def execute(self):
    self.log.info("Executing Transform script")
    while self.input.hasNext():
        try:
            # Read the next document, wrap it in a map and write out the wrapper
            in_doc = self.input.next()
            # wrapper = java.util.HashMap()
            # bucket is a property set in a mapper that precedes script snap and holds the bucket 
            # name
            bucket = in_doc.get("bucket")
            # $_bucketPipelineParam is a pipeline param and holds the bucket name
            bucketParam = $_bucketPipelineParam
            s3 = AmazonS3Client()
            usEast1 = Region.getRegion(Regions.US_EAST_1)
            s3.setRegion(usEast1)
            objectListing = s3.listObjects(ListObjectsRequest().withBucketName(bucket))
            s3objectList = {}
            for objectSummary in objectListing.getObjectSummaries():
                wrapper.put(objectSummary.getKey(),objectSummary.getSize())

self.output.write(in_doc, wrapper)
        except Exception as e:
            errWrapper = {
                'errMsg' : str(e.args)
            }
            self.log.error("Error in python script")
            self.error.write(errWrapper)

self.log.info("Finished executing the Transform script")

# The Script Snap will look for a ScriptHook object in the "hook"
# variable.  The snap will then call the hook's "execute" method.
hook = TransformScript(input, output, error, log)

# Script end

I am on windows so I’ve copied aws-java-sdk-1.9.6.jar file to c:/opt/snaplogic/userlibs on all of the plex nodes, you need to copy 3rd party jars to all of the plex nodes and make sure to save it at a consitent, same path on all nodes.
Also on my nodes I have edited/created a credentials file located here C:\Users\Bkukadia\.aws\credentials which contains these key=value pairs
aws_access_key_id=AWSKEYAKIAIGFUBXI
aws_secret_access_key=AWSSECRET8++Kg6QNMX6
I think you can also use IAM roles but I am not much familiar with it.
For more details on aws java sdk check this out - AWS SDK for Java
Invoke aws java sdk via py_2017_06_27.slp (6.0 KB)

Forum Discussion

How to use 3rd party python libraries in python Script

Recent Discussions

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?

Multipart Reader failure - 'content-type' was not found