integration with github is achieved via github rest api GitHub REST API - GitHub Docs
From design perspective this is how it works
Create a snaplogic pipeline that uses Meta Snaps - https://doc.snaplogic.com/wiki/display/SD/SnapLogic+Metadata+Snap+Pack get a list of SnapLogic assets (pipelines, tasks, files and accounts)
Invoke GitHub REST api (uses HTTP basic auth - https://doc.snaplogic.com/wiki/display/SD/Basic+Auth )
Read or Write to GitHub
Pipeline uses pipeline param to decouple runtime param from the actual implementation logic, so when you invoke these pipelines you can specify which Snaplogic projects to read, what assets to cin into GitHub, which repo to use on GitHub side and so on.
We have implemented bi-directional flow i.e. you can cin and cout source code from github
Attached SnapLogic project export has all the required files, please note that this is a custom solution, to use it you’ll need to keep your GitHub creds ready (repo name, uname and pwd), create a basic auth account in snaplogic and pass it on to the pipelines.
You may struggle a bit, but don’t give up, keep pounding and eventually you’ll crack it 🙂
Attached SnapLogic project export, please import it using these steps - https://doc.snaplogic.com/wiki/display/SD/How+to+Import+and+Export+Projects
Now for this
And that this can be further use to move code from one environment to another (code migration).
try this API
API Detail:
Syntax = https://elastic.snaplogic.com:443/api/1/rest/public/project/migrate/ORG/SPACE/PROJECT
Authorization Header = Basic Auth, pass on your Snaplogic uname/pwd
Body = application/json
Example:
https://elastic.snaplogic.com:443/api/1/rest/public/project/migrate/ConnectFasterInc/BK/DEV
{
"dest_path":"/tacobell/projects/bk",
"asset_types":["File","Job","Account","Pipeline"],
"async":"true",
"duplicate_check":"false"
}
Response:
{
“response_map”: {
“status_token”: “6e6600cd-2992-4423-95c3-ffb94293a3bd”,
“status_url”: “http://elastic.snaplogic.com/api/1/rest/public/project/migrate/6e6600cd-2992-4423-95c3-ffb94293a3bd”
},
“http_status_code”: 200
}
This runs as an async call and will migrate (copy) everything from ConnecFasterInc/BK/DEV to/tacobell/projects/bk, you can check status of the migration by visiting status_url
If a project already exists and duplicate_check set to false will create another project with the same name appended by (NUMBER) ex: if bk already exists inside /tacobell/projects then subsequent runs will add bk(1), bk(2) and so on, I wish we had an “overwrite” or “merge” parameter option but neverthless this is much easier than META snaps (IMO).
BK-Github Integration.zip (12.1 KB)
This is so helpful Bhavin! Thank you for jotting this down. I wil try this out and let you know how it goes.
I am pretty sure it will be helpful to us as well as larger community here on SnapLogic!
Many thanks for such a lot of detail and depth @Bhavin ! I will pass this onto our project team, a bit too intricate for me :).
Are there any plans to add native support for github/tfs into the product in the future? And integrations with TeamCity/Jenkins/etc?
Mike/Sudhendu, glad that you found it useful, these pipelines could be easily invoked via a jenkins / teamcity pipeline.
Before we jump into jenkins/teamcity or any ci-cd tool-chain we need to understand what kind of “artifacts” SnapLogic generates, at a high level you have a SnapLogic project which resides has a ORG/SPACE/PROJECT hierarchy where ORG is the tenant, space could be mapped to an ORG UNIT for ex: BI, DEV, any PROJECT-NAME and so on and with in each space you have more than one PROJECT.
When it comes to artifact you have
Pipelines
Jobs (Scheduled, Triggered, Ultra)
Accounts
Files (could be any anything, xml, json, script files, xslt , csv and so on which you your pipelines depends upon)
Pipelines are the actual work horse which are invoked via jobs, for lack of a better term pipelines are interpreted by the execution engine (JCC aka node) and hence there is not much to “build”, now with this architecture a typical CI-CD work flow may not directly fit neverthless I have seen customers would still like to leverage their fav CI/CD tool chain to automate as much as they can, tasks like
SnapLogic assets sharing via Github
Promoting projects from one ENV to another
Use Jenkins as the proverbial "rug" that ties the room together :)
Here is an actual implementation, we invoke SnapLogic pipelines (triggered tasks) via Jenkins job as an HTTP call.
Tools required
Pipeline design and Things to know
There are two projects involved
Project_Promotion = Utilizes Meta Snap pack to promote assets within same and different environments.
Github_Integration = Utilizes Meta Snap pack and REST Snap pack to promote assets within same and different environments. We are utilizing github rest api’s and they are invoked via basic_auth i.e. your github login account.
Pipelines:
Project_Promotion has
01 Main - Migrate Project
exposed as a triggered task
calls rest of the pipelines via pipeline execute
pipeline parameters
source_proj
target_proj
include_account
target_org
source_space
include_pipeline
update_task
target_space
update_account
include_task
source_org
Example values:
account_org = ConnectFasterInc
account_space = LCM
account_proj = Artifacts
source_org = ConnectFasterInc
source_space = LCM
source_proj = DEV
target_org = ConnectFasterPOC
target_space = BK
target_proj = PROD
GH_Owner = snapsrepo or your github account username
GH_Repo = reponame ex: cicd-demo
GH_Source_Path = relative path to repo ex: BK/DEV (case sensitive)
include_pipeline = true or false
include_account = true or false
include_task = true or false
update_account = true or false
Finished product would have a Jenkins pipeline that utilizes HTTP request plugin to invoke SnapLogic pipelines that read/write to GitHub and also promote projects from one env to another in SnapLogic. Using pipeline params we decouple source and target location along with what SnapLogic artifacts to “include” during Jenkins job invocation.
CICD-SnapLogic-Projects.zip (31.2 KB)
snaplogic-jenkins.docx (344.9 KB)
Wow this is very helpful.
I have been meaning to try to tackle this problem for over a year now, but never got around to it.
Now I know that it is possible.
Thanks,
TK
Hi @Bhavin,
The pipeline is really helpful and I managed to do a quick check.
However when I used the pipeline to import projects from Github to SL, I can see the assets being appended by extra values like,
Try Snaps-fce2f794-2e8e-48aa-90de-2cb1e762cb81
Instead of
Try Snaps
Do you know any obvious reason for this?
Am I missing any configuration?
/Krupali
Check the file writer somewhere which is appending the string
Try Snaps-fce2f794-2e8e-48aa-90de-2cb1e762cb81
“Try Snaps” must be appended in the beginning in the expression.
If I am guessing, this is the
‘Try Snaps’ + pipe.ruuid
Let us know if you cannot find it.
Thank you for the reply.
The issue is been solved, the last snap had for each asset had ‘+ “-” + pipe.ruuid’
like below,
_SL_Target_Proj + “/” + $property_map.info.label.value + “-” + pipe.ruuid
I removed that and it works as as expected. Thanks again.
Hello @bhavin.patel I tried to using above attached slps but I keep on getting below error.
"REST API service endpoint returned error result: status code = 404, reason phrase = Not Found, refer to the error_entity field in the error view document for more details
“error_entity”:
"{“message”:“Not Found”,“documentation_url”:"Repositories - GitHub Docs
{
“error”:
“REST API service endpoint returned error result: status code = 404, reason phrase = Not Found”
Steps I followed below :
Even from POST MAN app I also tried but no luck…
Appreciate your help here.
and tell me one more thing how differfent is if I want to do in gitlab instead github.
if you have any sample code for gitlab that will be very helpful for me.
@subhash.a.chandra Bhavin is no longer in the Community. Hopefully someone else can assist you.
Hello @dmiller I got the solution by myself and able to integrate Gitlab and Github both from SnapLogic.
Thanks for your info.
