Best practices to publish to a AWS SNS topic

[mohamadelmardin]

SnapLogic currently doesn’t have a dedicated snap pack for AWS SQS and SNS integration. Though we can use JMS consumer/producer snaps to configure it for SQS seamlessly however there is no mechanism to do that for SNS.

I guess the only possible way to publish to an SNS topic within a pipeline is use the REST api via GET/POST snaps. Is that true? What are the configuration steps required for security to send access and secret keys with the REST snap in order to do that? where should be keep these security tokens in the http parameter or http header? how can we further protect them?

Please provide best practices with working demo if any.

[aleung]

1.You can get access it here

2. You could try JS and Python as you describe as well.

Please let us know what ends up working for you.

[mohamadelmardin]

@aleung Just realized I never shared back my solution how I solved to publish to SNS topic directly from SnapLogic. I created this utility pipeline that leverage Python script to invoke AWS Java SDK to do the job.
I am setting the AWS secrets and other configuration such as topic name as pipeline parameters but they could be loaded from anywhere.

Attached is a PoC pipeline.
Publish to SNS Topic_2019_01_04.slp (7.1 KB)

I believe the AWS SDK are already installed on the groundplex node by default but if not you can download them from AWS site and drop them in the JVM library where the snaplex is running.

[hirakangshu_nag]

Hi Mohamed,

I had the same requirement with sns that you had posted. So if i follow the approach that you mentioned here it will be good right?
Or any additional step to be taken.

Regards,
Hirak

[KTsnap]

Hi,
We used the same approacj and it was working fine for past few months.But now am getting below error.

ailure: Cannot evaluate Script file: # Import the interface required by the Script snap. from com.snaplogic.scripting.language import ScriptHook import java.util from java.lang import System import sys sys.path.append(‘/opt/snaplogic/ext_jar/aws-java-sdk-1.11.788.jar’) from com.amazonaws.services.sns import AmazonSNSClient from com.amazonaws.regions import Region from com.amazonaws.regions import Regions from com.amazonaws import AmazonClientException from com.amazonaws import AmazonServiceException from com.amazonaws.services.sns.model import CreateTopicRequest from com.amazonaws.services.sns.model import CreateTopicResult from com.amazonaws.services.sns.model import SubscribeRequest from com.amazonaws.services.sns.model import PublishRequest from com.amazonaws.services.sns.model import PublishResult from com.amazonaws.services.sns.model import DeleteTopicRequest class TransformScript(ScriptHook): def init(self, input, output, error, log): self.input = input self.output = output self.error = error self.log = log # The “execute()” method is called once when the pipeline is started # and allowed to process its inputs or just send data to its outputs. def execute(self): self.log.info(“Executing Transform script”) # Specify security credentials explecitly props = System.getProperties() props.setProperty(“aws.accessKeyId”, “”) props.setProperty(“aws.secretKey”, “”) # create a new SNS client and set endpoint snsClient = AmazonSNSClient() snsClient.setRegion(Region.getRegion(Regions.US_WEST_2)) while self.input.hasNext(): try: # Read the next document, wrap it in a map and write out the wrapper in_doc = self.input.next() # Specify security credentials explecitly props = System.getProperties() props.setProperty(“aws.accessKeyId”, in_doc[‘accessKeyId’]) props.setProperty(“aws.secretKey”, in_doc[‘secretKey’]) # create a new SNS client and set endpoint snsClient = AmazonSNSClient() snsClient.setRegion(Region.getRegion(Regions.US_WEST_2)) # publish to an SNS topic publishRequest = PublishRequest(in_doc[‘topicArn’], in_doc[‘msg’]) publishResult = snsClient.publish(publishRequest) #Put the incoming message in a wrapper object to write it to output stream wrapper = java.util.HashMap() wrapper[‘original’] = in_doc self.output.write(in_doc, wrapper) except Exception as e: errWrapper = { ‘errMsg’ : str(e.args) } self.log.error(“Error in python script”) self.error.write(errWrapper) self.log.info(“Finished executing the Transform script”) # The Script Snap will look for a ScriptHook object in the “hook” # variable. The snap will then call the hook’s “execute” method. hook = TransformScript(input, output, error, log), Reason: ImportError: No module named sns in at line number 8, Resolution: Please fix the script file error at line: 8 column: -1

[tochen_snap]

Hello Aaleung, Where is the “showcase” that you are talking about? I have hard time to find it. thx