Using Github as a code repository for SnapLogic artifacts

integration with github is achieved via github rest api GitHub REST API - GitHub Docs
From design perspective this is how it works

Create a snaplogic pipeline that uses Meta Snaps - https://doc.snaplogic.com/wiki/display/SD/SnapLogic+Metadata+Snap+Pack get a list of SnapLogic assets (pipelines, tasks, files and accounts)

Invoke GitHub REST api (uses HTTP basic auth - https://doc.snaplogic.com/wiki/display/SD/Basic+Auth )

Read or Write to GitHub

Pipeline uses pipeline param to decouple runtime param from the actual implementation logic, so when you invoke these pipelines you can specify which Snaplogic projects to read, what assets to cin into GitHub, which repo to use on GitHub side and so on.

We have implemented bi-directional flow i.e. you can cin and cout source code from github

Attached SnapLogic project export has all the required files, please note that this is a custom solution, to use it you’ll need to keep your GitHub creds ready (repo name, uname and pwd), create a basic auth account in snaplogic and pass it on to the pipelines.

You may struggle a bit, but don’t give up, keep pounding and eventually you’ll crack it 🙂

Attached SnapLogic project export, please import it using these steps - https://doc.snaplogic.com/wiki/display/SD/How+to+Import+and+Export+Projects

Now for this

And that this can be further use to move code from one environment to another (code migration).

try this API

API Detail:

Syntax = https://elastic.snaplogic.com:443/api/1/rest/public/project/migrate/ORG/SPACE/PROJECT
Authorization Header = Basic Auth, pass on your Snaplogic uname/pwd
Body = application/json

Example:

https://elastic.snaplogic.com:443/api/1/rest/public/project/migrate/ConnectFasterInc/BK/DEV

{

           "dest_path":"/tacobell/projects/bk",
           "asset_types":["File","Job","Account","Pipeline"],
           "async":"true",
           "duplicate_check":"false"

}

Response:

{
“response_map”: {
“status_token”: “6e6600cd-2992-4423-95c3-ffb94293a3bd”,
“status_url”: “http://elastic.snaplogic.com/api/1/rest/public/project/migrate/6e6600cd-2992-4423-95c3-ffb94293a3bd”
},
“http_status_code”: 200
}

This runs as an async call and will migrate (copy) everything from ConnecFasterInc/BK/DEV to/tacobell/projects/bk, you can check status of the migration by visiting status_url
If a project already exists and duplicate_check set to false will create another project with the same name appended by (NUMBER) ex: if bk already exists inside /tacobell/projects then subsequent runs will add bk(1), bk(2) and so on, I wish we had an “overwrite” or “merge” parameter option but neverthless this is much easier than META snaps (IMO).

BK-Github Integration.zip (12.1 KB)

I think @Bhavin may have some advice here as his LinkedIn says:

• Prototyped integrating SnapLogic pipelines with CI/CD tool chain (ex: Jenkins/TeamCity)

(sorry, I like a google search ;))

Am also interested in this as we spend a lot of time and expertise designing and testing pipelines (as well as ad-hoc workflows) and to rely on manual export with no change history/backup/CI is not ideal. We’re using both TeamCity and Jenkins (for CI and CD) on various other platforms with github as our source code repo. Any integration/scripts/plugins/webhooks for SL snaplexes would be great.

(@nganapathiraju I see you posted on the TFS thread that customers are integrating with github - searching “github” in the documentation area has 0 matches, are there some whitepapers/guidance on how to integrate SL with github?)

So the solution provided here is just an asset “copy”.
Meaning that it simply copies the original asset from the source place and copy it to the destination place.
The github is just a place to store the location of the original assets.

The promotion pipeline will read the files in github to get the location of the original asset and move(copy) it to destination location.

The problem of this is that, the files store in Github is not being used as source code. If I remove the asset from the source location, then the promotion pipeline will not be able to migrate the asset from source to destination as its doing a copy and the asset is missing.

Is there a way to take the files store in github as true source code and import this source code to a new org? In this case, even someone removes all the assets from source org, by importing the files in Git, we should still be able to migrate all the assets into destination org.

Thanks

Hi,

Note I have also shared our current solution for source control in the other thread: GITHUB integration.

Cheers,
C.J.

this is working fine when dealing with default branch… but how do i work with specific branch ?

Anyone tried to migrate project from GIT tag to a different organization that the current one where the pipeline is running and migrate the files to the upper organization(example prod)?

Succeeded migrating everything except files if they are stored on GIT and the migration pipeline from DEV org to PROD org is on DEV, then the pipeline is not migrating(creating) the files on the upper org.

Hi All , I am stuck there in the jenkins project while trying to create the pipeline for Snaplogic task .Can anyone please tell me the steps for this in Jenkins ? First it will be one Jenkins Pieline , right ? or it will be Jenkins project ?

Thanks,
Rudradip