PSAmmirata
Employee
Joined 9 years ago
User Profile
User Widgets
Contributions
Re: AWS S3 Account "Cross Account IAM" benefit over using resource-based policy?
Hi @sanjaynayak - Since the IAM role is attached to the ec2 instances hosting the snaplex nodes, I believe you can control access within SnapLogic at the project level by whether or not you create the SnapLogic account.2.5KViews0likes0CommentsSnaplex-based scheduler for scheduled tasks - how to know when snaplex node is synchronized with control plane?
We have a two-node Groundplex. The nodes are Linux (RHEL). Every month we patch the OS on these nodes. We have written the following scripts to help facilitate the patching process Script to enter maintenance mode: Use Snaplex Management API to put node into maintenance mode. Call Snaplex Monitoring API to get the status of the node. Keep calling the API until the node status is not “running”. Call Pipeline Monitoring API to get the list of “running” pipelines. Keep calling the API until there are no running pipelines. Script that we use when exiting maintenance mode: Call Snaplex Monitoring API to get the status of the node. Keep calling the API until the node status = down or running. (We do this to ensure that the node has restarted after the reboot). Use Snaplex Management API to take the node out of maintenance mode. Call Snaplex Monitoring API to get the status of the node. Keep calling the API until the node status = “running”. The overall patching process is executed by the Linux admin and goes like this: Invoke script to put node1 into maintenance mode. Apply OS patch to node1 and reboot node1. SnapLogic automatically starts on node1 after the reboot because we have it configured to do so using the init.d utility. Invoke script to take node 1 out of maintenance mode. As soon as node 1 comes up the script to put node2 into maintenance mode is invoked. Apply OS patch to node2 and reboot node2. SnapLogic automaticaly starts on node2 after the reboot because we have it configured to do so using the init.d utility. Invoke script to take node 2 out of maintenance mode. We executed this patching process and node1 was rebooted and the JCC restarted at 8/1 5:56:53PM. node2 was rebooted and the JCC restarted at 8/1 6:01:03PM. We have three test scheduled tasks that are scheduled to run every fifteen minutes. We observed the following behavior: test_scheduler_1 task - executed at 5:45PM as expected, but didn’t execute per the schedule again until the 6:30 schedule (6:44). Two scheduled events were missed (6:00 and 6:15). test_scheduler_2 task - executed at 5:45PM as expected, but didn’t execute per the schedule again until the 7:30 schedule (7:37). Six scheduled events were missed (6:00, 6:15, 6:30, 6:45, 7:00, and 7:15). test_scheduler_3 task - executed at 5:45PM as expected, but didn’t execute per the schedule again until the 7:00 schedule (7:01). Four scheduled events were missed (6:00, 6:15, 6:30, and 6:45). Even though we always had a node up and running in the Snaplex during the patching process, since both nodes were rebooted, it appears that scheduled events were missed because the scheduling information had not been pushed from the control plane to the snaplex nodes. If this is the case, is there an API that we can use to determine the state of the snaplex node scheduling information? We’d like to modify our patching scripts so that we don’t start patching node2 until the JCC on node1 is up AND the scheduling information on that node is synchronized with the control plane. Can you suggest such a way to do that?2.8KViews1like2CommentsRedshift Bulk Upload - Cross Account IAM role support?
Our SnapLogic snaplex node resides in AWS Account A. Our Redshift cluster and S3 bucket reside in AWS Account B. I configured the IAM roles as follows: AWS Account B - created an IAM role that trusts Account A and allows s3::GetObject, s3::PutObject, and s3::DeleteObject on the S3 bucket in Account B. AWS Account A - created an IAM role that’s attached to the ec2 instance that hosts the Snaplogic snaplex node. This IAM role is allowed to assume the role created in #1 (above). AWS Account B - created another IAM role that allows s3::GetObject, s3::PutObject, and s3::DeleteObject on the S3 bucket in Account B. This role is attached to the Redshift cluster. The Redshift Cross-Account IAM Role Account is configured as follows: Account properties/Cross-Account IAM Role ARN (writing to Bucket) is configured with the arn for the IAM role created in #1 (above). IAM properties (Redshift Cluster)/IAM role name is configured with the role name for the IAM role created in #3 (above). Cross-Account IAM properties (S3 Bucket) is configured with the arn for the IAM role created in #1 (above). The Redshift Bulk Upsert snap is configured with ‘IAM role’ checked. When executing the pipeline with the Redshift Bulk Upsert snap I can see that data is written to S3, and then the snap fails with an error that states that the Redshift user is not authorized to assume the IAM roles (#1 and #3 from above). User arn:aws:redshift:region:accountb:db_user:redshift/user is not authorized to assume IAM Role arn:aws:iam::accountb:role/role_3,arn:aws:iam::acountb:role/role_1 Even though best practice would be for the Redshift cluster and the S3 bucket to be in the same AWS account, does SnapLogic’s cross account IAM role support allow for the Redshift cluster and the S3 bucket to reside in different AWS accounts? If so, is this why the Redshift user is attempting to assume the IAM role even though the Redshift cluster and the S3 bucket reside in the same AWS account and the IAM role is already attached to the Redshift cluster? https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1246956316/Configuring+Cross+Account+IAM+Role+Support2.8KViews0likes3CommentsRedshift Bulk Upsert - how does it work?
How does the Redshift Bulk Upsert work underneath the covers? For example, I know that the Redshift Bulk Load snap writes the data to S3 and then executes a Redshift COPY command to load the data from S3 to the Redshift table. How does the Redshift Bulk Upsert work?2.3KViews0likes2CommentsRe: Generic JDBC - interaction between exhaustion of connection pool and 'number of retries' setting?
Thanks @mbowen. So, for a “read” operation that times out on the initial connection attempt it won’t go into the retry block. Correct? If the “read” operation fails for some other reason will it go into the retry block? Or does the “read” operation never use the retry block?2.6KViews0likes1CommentGeneric JDBC - interaction between exhaustion of connection pool and 'number of retries' setting?
Let’s say I have a Generic JDBC account configured with MaxPoolSize= 10 connections and Checkout Timeout - 10,000 milliseconds. I have a Generic JDBC Execute snap configured with ‘number of retries’ = 5 and ‘retry interval’ = 10 seconds. When the first JSON document flows into the Generic JDBC Execute snap, it will tries to obtain a connection from the connection pool, but there are already 10 connection in the connection pool and they’re all in use. The Snap waits 10,000 milliseconds for a connection to become free, but no connection comes free during that time. Does the Snap fail at this time? OR, will the snap wait the 10 second ‘retry interval’ and try to obtain a connection from the connection pool again?3KViews0likes4CommentsJDBC connection pooling - when are minimum number of connections created?
If I have a Generic JDBC Account configured with a minimum pool size > 0, when does the connection pool get created and initialized with the minimum number of connections? Is it when the first pipeline that uses the Generic JDBC Account is being prepared/started before any documents flow into a Snap that uses the Generic JDBC Account? Or does the connection pool only get created and initialized with the minimum number of connections at the time that the first document flows into a Snap that uses the Generic JDBC Account? The reason for asking the question is that we have an Ultra task pipeline with a Snap that uses a Generic JDBC Account. The minimum pool size is set to 0 (zero). What we found is that the first HTTP request made to the Ultra task times out, but subsequent requests don’t time out. I believe that since we have the minimum pool size set to 0 (zero) that the connection isn’t created until the first document flows into the Snap that uses the Generic JDBC Account. Making that connection takes too much time and the result is that the HTTP request times out. I want to confirm that if we set the minimum pool size > 0 that the connection will be made before the first document flows into the Snap that uses the Generic JDBC Account.2.1KViews0likes1Comment