Migrate projects, pipelines, accounts, tasks across environments

[dshen]

One of the common requests from customers is how to move Snaplogic asset (pipelines, accounts and tasks) in a project from one environment to another easily as part of the configuration management or promotion processes. The standard project export/import function will move the pipeline, tasks, files, etc., but the customer will need to re-create the account objects and re-link all the account references in the target environment. This extra step could be a hassle if you have a big project with many pipelines and snaps require account references.

The sample pipeline here use Snaplogic Metadata Snapack to read the asset definition from a project and write to the target location (same org or not). It will move the accounts, pipelines and tasks from source to target. More importantly, it will maintain the account references in the pipeline and pipeline reference in the task. User just need to re-enter the Account password and re-select the Task snaplex in the target environment.

The attached project export contains 4 pipelines:

1. 01 Main - Migrate Project: This is the main pipeline which will call into the following in sequence to move the assets.
2. 02 Upsert Account - This pipeline will move the Account object in a project to the target location.
3. 03 Upsert Pipeline - This pipeline will move the Pipelines in a project to the target location.
4. 04 Upsert Task - This pipeline will move the Tasks object in a project to the target location.

User can specify the source and target location (org + space + project) in the pipeline parameters. To run the pipeline to move project across org, the user account will need to have read/write permission to both source and target location.

The project export of those pipelines can be downloaded here:

Migrate Project v2.zip (10.7 KB)

[jason_cotterell]

@psadasivam I wish the professional services that we paid to come on site had this sort of knowledge. Now we’re in a lurch.

[Ksivagurunathan]

@psadasivam Are you comparing your GIT pipelines with existing pipelines before updating them on the target location ? How are you comparing the pipelines

[psadasivam]

@Ksivagurunathan - Nope, We don’t do any development in http://uat.elastic.snaplogic.com . Our pipelines will always overwrite the existing pipeline version.

[Ksivagurunathan]

Just curious, you deploy every pipelines all the times or you maintain a list of pipelines to be deployed every release.

[MuraliNagula]

While moving the pipelines the associated account (redshift) needs to be in the same project space ? Our pipelines use the account from shared folder at the top level. Although the account name/label is same the account is not being populated while using the migrate pipelines mentioned in this post. Is this expected and are there any workarounds ?