Migrate projects, pipelines, accounts, tasks across environments

One of the common requests from customers is how to move Snaplogic asset (pipelines, accounts and tasks) in a project from one environment to another easily as part of the configuration management or promotion processes. The standard project export/import function will move the pipeline, tasks, files, etc., but the customer will need to re-create the account objects and re-link all the account references in the target environment. This extra step could be a hassle if you have a big project with many pipelines and snaps require account references.

The sample pipeline here use Snaplogic Metadata Snapack to read the asset definition from a project and write to the target location (same org or not). It will move the accounts, pipelines and tasks from source to target. More importantly, it will maintain the account references in the pipeline and pipeline reference in the task. User just need to re-enter the Account password and re-select the Task snaplex in the target environment.

The attached project export contains 4 pipelines:

  1. 01 Main - Migrate Project: This is the main pipeline which will call into the following in sequence to move the assets.
  2. 02 Upsert Account - This pipeline will move the Account object in a project to the target location.
  3. 03 Upsert Pipeline - This pipeline will move the Pipelines in a project to the target location.
  4. 04 Upsert Task - This pipeline will move the Tasks object in a project to the target location.

User can specify the source and target location (org + space + project) in the pipeline parameters. To run the pipeline to move project across org, the user account will need to have read/write permission to both source and target location.

The project export of those pipelines can be downloaded here:

Migrate Project v2.zip (10.7 KB)

4 Likes

the attachment doesn’t contain .slb file needed to import into snapLogic

There isn’t an .slb file format.
Individual pipelines are exported out as .slp files.
Projects are exported out as .zip files and imported in as such within Manager. Once you import the project, you should see the 4 pipelines.

We are able to use this to move stuff from our Stage or to our Prod org (and reversed) but we do not seem to be able to move to our UAT environment, which is where we need to test our pipeline against the forced upgrade November 10th.

When we try, we get:
The requested resource at the supplied path does not exist: /SVG-UAT/projects

Any word on how to move resources to the UAT environment?

@jason.cotterell, I don’t believe you can migrate code between http://elastic.snaplogic.com and http://uat.elastic.snaplogic.com. We push all our assets to Git from elastic.snaplogic.com and pull from Git to uat.elastic.snaplogic.com using rest/metadata snaps.
Hope this helps.

@psadasivam I wish the professional services that we paid to come on site had this sort of knowledge. Now we’re in a lurch.

Is there any way to export as slp files without downloading the pipeline file?

Basically I am looking for any automate way of export the entire/specific set of pipelines to S3 or other storage for backup.

I don’t need any manual kind of import and export works? I need any best approach to get the slp file by automation.

Please provide any suggestions? Thanks in advance.

Am developing a pipeline which can be used for Check out, check in to GIT and migrating the pipelines/task/account from GIT(not GIT HUB) to Snaplogic Prod org. I have created a Service account in snaplogic. I need to run the migration pipelines using the service account, so that any pipelines/task/account created in production snaplogic org is owned by service account user. How do I achieve that ?

Mostly likely I will call migration pipeline via triggered task from ansible/Jenkins pipelines.

How are you all migrating accounts from GIT. The account config will be different for dev/qa/prod

@dshen - Can you help answer the questions ?

@psadasivam Are you comparing your GIT pipelines with existing pipelines before updating them on the target location ? How are you comparing the pipelines

@Ksivagurunathan - Nope, We don’t do any development in http://uat.elastic.snaplogic.com . Our pipelines will always overwrite the existing pipeline version.

Just curious, you deploy every pipelines all the times or you maintain a list of pipelines to be deployed every release.