Using Github as a code repository for SnapLogic artifacts

[sudhendu]

Hello SL community!!

Coming from a typical SOA/ESB background and working on tools like TIBCO Businessworks, Webmethods, etc. I was wondering if we have a way of using any code repository tool (Github, etc.) to store our Projects and other artifacts.

I understand we have a intuitive Import/Export capabilities, this is manual and is a explicit activity to be performed by a developer.

I was wondering:

1. if we have anything (or is there a possibility) to integrate SL and GitHub.
   This should ensure that the Project and its assets gets checked-in into the GitHub repo without us doing any manual import-export.

2. And that this can be further use to move code from one environment to another (code migration).

I am new to the SL world, but already finding the community really helpful 🙂

All response would be of help 🙂

Thank you!
Sudhendu

[mikeandrews]

I think @Bhavin may have some advice here as his LinkedIn says:

• Prototyped integrating SnapLogic pipelines with CI/CD tool chain (ex: Jenkins/TeamCity)

(sorry, I like a google search ;))

Am also interested in this as we spend a lot of time and expertise designing and testing pipelines (as well as ad-hoc workflows) and to rely on manual export with no change history/backup/CI is not ideal. We’re using both TeamCity and Jenkins (for CI and CD) on various other platforms with github as our source code repo. Any integration/scripts/plugins/webhooks for SL snaplexes would be great.

(@nganapathiraju I see you posted on the TFS thread that customers are integrating with github - searching “github” in the documentation area has 0 matches, are there some whitepapers/guidance on how to integrate SL with github?)

[Bhavin]

integration with github is achieved via github rest api GitHub REST API - GitHub Docs
From design perspective this is how it works

Create a snaplogic pipeline that uses Meta Snaps - https://doc.snaplogic.com/wiki/display/SD/SnapLogic+Metadata+Snap+Pack get a list of SnapLogic assets (pipelines, tasks, files and accounts)

Invoke GitHub REST api (uses HTTP basic auth - https://doc.snaplogic.com/wiki/display/SD/Basic+Auth )

Read or Write to GitHub

Pipeline uses pipeline param to decouple runtime param from the actual implementation logic, so when you invoke these pipelines you can specify which Snaplogic projects to read, what assets to cin into GitHub, which repo to use on GitHub side and so on.

We have implemented bi-directional flow i.e. you can cin and cout source code from github

Attached SnapLogic project export has all the required files, please note that this is a custom solution, to use it you’ll need to keep your GitHub creds ready (repo name, uname and pwd), create a basic auth account in snaplogic and pass it on to the pipelines.

You may struggle a bit, but don’t give up, keep pounding and eventually you’ll crack it 🙂

Attached SnapLogic project export, please import it using these steps - https://doc.snaplogic.com/wiki/display/SD/How+to+Import+and+Export+Projects

Now for this

And that this can be further use to move code from one environment to another (code migration).

try this API

API Detail:

Syntax = https://elastic.snaplogic.com:443/api/1/rest/public/project/migrate/ORG/SPACE/PROJECT
Authorization Header = Basic Auth, pass on your Snaplogic uname/pwd
Body = application/json

Example:

https://elastic.snaplogic.com:443/api/1/rest/public/project/migrate/ConnectFasterInc/BK/DEV

{

           "dest_path":"/tacobell/projects/bk",
           "asset_types":["File","Job","Account","Pipeline"],
           "async":"true",
           "duplicate_check":"false"

}

Response:

{
“response_map”: {
“status_token”: “6e6600cd-2992-4423-95c3-ffb94293a3bd”,
“status_url”: “http://elastic.snaplogic.com/api/1/rest/public/project/migrate/6e6600cd-2992-4423-95c3-ffb94293a3bd”
},
“http_status_code”: 200
}

This runs as an async call and will migrate (copy) everything from ConnecFasterInc/BK/DEV to/tacobell/projects/bk, you can check status of the migration by visiting status_url
If a project already exists and duplicate_check set to false will create another project with the same name appended by (NUMBER) ex: if bk already exists inside /tacobell/projects then subsequent runs will add bk(1), bk(2) and so on, I wish we had an “overwrite” or “merge” parameter option but neverthless this is much easier than META snaps (IMO).

BK-Github Integration.zip (12.1 KB)

[sudhendu]

This is so helpful Bhavin! Thank you for jotting this down. I wil try this out and let you know how it goes.

I am pretty sure it will be helpful to us as well as larger community here on SnapLogic!

[mikeandrews]

Many thanks for such a lot of detail and depth @Bhavin ! I will pass this onto our project team, a bit too intricate for me :).

Are there any plans to add native support for github/tfs into the product in the future? And integrations with TeamCity/Jenkins/etc?