GITHUB integration

Community!

I wanted to see what the interest was around github integration and what exactly that means to each of you. This is probably best as a private 1-1 conversation so please reply to this thread and I will reach out directly for follow up.

Thanks!
-Tim

6 Likes

You mean Snaplogic Pipelines integration with GIT in the UI or GIT Snaps ?

Great question! Integrations with GIT in the UI. More so using GIT as a means of versioning org assets.

1 Like

We have already requested for that enhancement with snaplogic. I can get into a call if required to discuss more on this. We use GIT for version control code changes in our company.

Perfect, I will reach out in future to discuss further.

If anybody else has interest, chime in!

Thanks!
-Tim

We started by leveraging pipeline created by Bhavin. It will be great a value add if SnapLogic is planning for an OOTB feature

Best,
Yashu Vyas

GitHub integration would be a very important feature and would add lot of value.

In the current environment, we have to keep track of what pipelines have been changed and promote them to the next environment. Versioning is also not automatic.

We have built some automation to export the pipeline and add it to our GIT repository. It still needs us to run it keep track of the pipelines we have changed, and run the script on them.

I can definitely get on a call to discuss any questions you may have

We’re also very interested in native GitHub integration. Our internal processes require deployments from GitHub but we’ve had to develop our own sets of pipelines leveraging the metadata and REST snaps in order to make it manageable.

Awesome! I will reach out and continue to reach out to anybody who responds to this thread.

Regards,
-Tim

We also use git for version control and are very interested in being able to do with with our pipelines as well.

We are very interested in using GitHub as an artifact/revision repository.

I have been meaning to look into some of the bespoke solutions mentioned in the forum, but just haven’t had the time.

Hence, a built-in integration would be best.

TK

@tlui, We are more interested to have GIT integration for SnapLogic assets.

Let me if you need additional details.

Hi,

Our company basically needs this feature badly which to align our company development rule.

So by looking at Bhavin’s post and actually we got a same solution from service professional.

So the solution provided here is just an asset “copy”.
Meaning that it simply copies the original asset from the source place and copy it to the destination place.
The github is just a place to store the location of the original assets.

The promotion pipeline will read the files in github to get the location of the original asset and move(copy) it to destination location.

The problem of this is that, the files store in Github is not being used as source code. If I remove the asset from the source location, then the promotion pipeline will not be able to migrate the asset from source to destination as its doing a copy and the asset is missing.

So a ‘real’ CICD solution is needed from Snaplogic pespective to make this as a more modern development tool.

Thanks

I’m definitely interested in being able to extract the pipelines and store it in our local GIT repo and use that process for source control and migration to go along with project deployments. I was just looking snaplogic read snap and how to start to pull this out and then I decided to come to the community. :slight_smile:

We would LOVE Git integration! We’ve actually been considering building something like this ourselves. This would be very valuable to us!

Definitely code versioning on GIT is needed.

It should be noted that Git integration and Github integration are two slightly different things.

I’ve worked and shared with Bhavin to create a Github integration we’re using at our company. I have a flavor that handles tagging branches for versioning. We also have integration with Teamcity working so that we can deploy to different Snaplogic orgs from Github source using a TeamCity project.

A main difference between Bhavin’s work and my own is that ours uses Github API Access Tokens to gain access to the necessary API since we use private repos and not public. Ours also uses Blobs to store snaplogic metadata/code so it doesn’t have the size limitations you might otherwise run into. Another difference is that we handle files, pipelines, tasks and accounts. We use a manifest file to cherry pick which files we care to store into Github. Also we create a version.txt file with each commit to store the commit information someplace to verify the version of Snaplogic code that is in a commit and is currently deployed in an org.

We’ve been successfully using the code to store our Snaplogic metadata/code and to deploy changes from org to org for over six months now.

I can try to see if my manager would be ok with me releasing our Github integration and deployment code to a public repo if there’s interest.

Charley Lee
sunlee@soundphysicians.com
Senior Data Integration Developer
Sound Physicians

1 Like

it would be great if you could share this

Hi,

Source control integration is of great interest to us too, so please include us in any discussions. In my opinion, having source control out of the box is currently one of the major gaps in Snaplogic which I’d expect of an enterprise-level tool.

Note that we also have an additional requirement - integration with on-premise BitBucket (Stash). Although this uses git under the covers, there is currently no REST API so the solutions for GitHub don’t work for us.

We’ve developed an interim process using a SFTP server as a proxy for source repository. Although Snaplogic have recently added public APIs for promotion of artefacts between orgs, we didn’t want to use these - as mentioned above, we wanted the source repository to be the source of truth for promotions, rather than promoting code directly from DEV to TEST and PROD.

Here is our current process:

  • We have a pipeline called “Push Assets” in our DEV org which:
    – Exports all Pipelines, Files, Accounts and Tasks to the DEV folder (“branch”) in our source control (SFTP)
    – Exports artefacts with filename in format {name}.{updated_datetime_stamp}.{type} in a folder structure according to the Project Space / Project it came from
    – Each time the export is run, a new datetime stamped folder is created, so we have a history of all past versions
    – Runs daily or can be run manually by any developer
    – Has a parameter to select a specific Project Space, or by default exports all Project Spaces
  • To promote artefacts, developers log into source control (SFTP) and copy the relevant artefacts into the TEST or PROD folders / “branches” in the corresponding Project Space / Project folders
  • We then have a pipeline called “Promote Assets” which runs in our PROD org
    – This requires permissions on all orgs / Project Spaces / Projects so runs as an admin, therefore it can’t be triggered by a developer and instead runs on a scheduled basis every 20 mins
    – This only promotes Pipelines and Tasks - Accounts need to be created per environment due to how Snaplogic stores credentials and are created rarely, so we create these manually, and Files we would like to promote but haven’t found a way to import files cross-org yet (working on it)
    – It looks in the TEST and PROD folders / “branches” in the source repository, compares the date time stamp on the file to the updated date / time on the corresponding artefact in the target org (if it exists) and imports the artefact if it is newer
    – If it is creating a new scheduled Task, we default it to disabled in the target org, so that it doesn’t start running before we are ready

In this way, TEST and PROD are always representative of what has been “merged” into the TEST and PROD “branches” (folders) in the source repository. We hope in the future to replace the SFTP solution with BitBucket (Stash).

The process also has a whole heap of magic to ensure the integrity of artefacts (links to Accounts, Pipeline Execute links to Pipelines, remapping Snaplex names per environment, etc.) - don’t underestimate this effort!

Note we also use this process for promoting code to our UAT (platform release testing) environment, so we can easily test new platform releases.

Hope this was useful and look forward to enhancing the Snaplogic source control further.

Cheers,
C.J.

2 Likes

please share the project