Best practices to publish to a AWS SNS topic

SnapLogic currently doesn’t have a dedicated snap pack for AWS SQS and SNS integration. Though we can use JMS consumer/producer snaps to configure it for SQS seamlessly however there is no mechanism to do that for SNS.

I guess the only possible way to publish to an SNS topic within a pipeline is use the REST api via GET/POST snaps. Is that true? What are the configuration steps required for security to send access and secret keys with the REST snap in order to do that? where should be keep these security tokens in the http parameter or http header? how can we further protect them?

Please provide best practices with working demo if any.

I have shared sample pipelines on the “Showcase” section. You should be able to find the pipeline under Endpoints / AWS SQS.

You could use the JMS Producer to send message to AWS SQS and JMS Consumer to retrieve AWS SNS messages.

Thank you @aleung but this doesn’t address my issue and question. I don’t have a problem publishing and consuming messages from SQS. My problem is only with publishing to SNS topic since there isn’t snap to handle SNS topic publishing. I was wondering if there is a workaround either by using REST or javascript snaps to achieve that purpose.

In that case you need to get AWS CLI and python install it in your groundplex. Once you have this setup, you could try using “Execute Command Line” under Utilties folder in Box to execute: aws sns publish.

Reference: https://docs.aws.amazon.com/cli/latest/reference/sns/publish.html

@aleung

So you are referring me to use the CLI AWS SDK to integrate with SNS at runtime I have two questions there:

  1. I was able to download and run commands with AWS CLI on my local desktop machine but not sure how I would execute this command from my pipeline and after I set up and download CLI on groundplex. You mentioned Utilities folder in Box but were can I find it? Are you referring to Box (online file sharing service) snap pack or something else that I am completely out of context on? Also where is that Utilities folder located?

  2. Can I use another AWS SDK instead of CLI such as JavaScript or Python? The reason I mentioned these two other SDKs because I could leverage a script snap that supports any of these two languages and call the SNS API there. I am fine with the CLI if it works but I am just hinting for other suggestions if possible.

1.You can get access it here

  1. You could try JS and Python as you describe as well.

Please let us know what ends up working for you.

Hello Aaleung, Where is the “showcase” that you are talking about? I have hard time to find it. thx

https://community.snaplogic.com/c/pipelines/showcase

The link above is the location of the showcase and you can locate " Box’s Snaplogic Knowledge Base" then it is in the following location.

SnapLogic - Knowledge Base > Endpoints > AWS SQS

@aleung Just realized I never shared back my solution how I solved to publish to SNS topic directly from SnapLogic. I created this utility pipeline that leverage Python script to invoke AWS Java SDK to do the job.
I am setting the AWS secrets and other configuration such as topic name as pipeline parameters but they could be loaded from anywhere.

Attached is a PoC pipeline.
Publish to SNS Topic_2019_01_04.slp (7.1 KB)

I believe the AWS SDK are already installed on the groundplex node by default but if not you can download them from AWS site and drop them in the JVM library where the snaplex is running.