dwhansen-cbg
Contributor
Joined 6 years ago
User Profile
User Widgets
Contributions
Email admin on all pipeline failures
Over the weekend we had a very important pipeline fail which has caused other processes we have to fail. Our SnapLogic process is designed to notify our system admin any time a pipeline fails by using an Error pipeline with an email snap in it. The problem is that this doesn't work for all scenarios. The reason the pipeline failed was due to the error: Pipeline is no longer running Reason: Lost contact with Snaplex node while the pipeline was running This made it so the Error pipeline was never started and we never got a notification that this pipeline failed. My question is this: is there any other way to be notified of ANY pipeline failure regardless of how the pipeline fails? I've looked through all the settings and I'm currently not seeing anything.1.1KViews0likes0CommentsCloudplex logging to Datadog
We run our development and production environments on a Cloudplex. We would like to send all logs to Datadog. Is there a way we can do this? Is there an integration with Datadog? I understand that it would be relatively simple to do this on a Cloudplex, but we don’t want to manage the hardware.1.9KViews0likes1CommentRe: Redshift Bulk Load on Cloudplex
We think we figured out what was happening. Every 180 days we are required by our company policies to swap out our Access Keys and Secrets for security purposes. When we swapped them out, there was somehow a value in the Token field on the Redshift Account. We tried again by updating the Access Key and Secret, then clearing out the Token field and that seemed to do the trick. We’re not sure if this is a long-term fix or if it will fail again after a certain amount of time, but we are good to go for now.7.2KViews1like0CommentsRe: Redshift Bulk Load on Cloudplex
@ptaylor It’s possible I’m misunderstanding what the token is and how to get it, but my current understanding is that we need to get an AWS STS Token using either the AWS CLI or SDK. I’m basing my statement on what is in this documentation: get-session-token — AWS CLI 1.27.51 Command Reference This duration can range from 900 seconds (15 minutes) up to a maximum of 129,600 seconds (36 hours), with a default of 43,200 seconds (12 hours). Can you point me in the right direction on how to get a token (how you’re doing it so there isn’t a time-based limitation) so we can use the Bulk Load snap?4.3KViews0likes1CommentRedshift Bulk Load on Cloudplex
We have a Redshift cluster that we would like to load data into. Using the Redshift - Insert snap is simple, but prohibitively slow and isn’t the recommended way to load data into a Redshift. We would like to use the Redshift - Bulk Load snap, but are running into a few issues when setting up the Redshift Account in SnapLogic. Our understanding is that using IAM Roles for authentication is NOT possible on a Cloudplex. Is this true? If so, this is a huge issue. If we can’t use an IAM Role for authentication, the only other option is an AWS Access Key with its secret and token. The main issue with this approach is that the tokens are temporary and only last a few hours at a maximum. How can we use an AWS Access Key with its secret and token without having to refresh the token every 15 minutes? This doesn’t seem useful. Any help would be great. Thanks!Solved7.4KViews0likes9Comments