GitHub Environmental Design

[CookieMonster]

Hi,

We’re a new SL user and are wrestling with the best way to join up our SL projects and repositories/branches in GitHub. There have been some posts on this previously but nothing recent we could see, so we wondering how people have set this up since the GitHub Integration was released.

We would ideally like something that allows multiple streams of work to be developed and then brought together at the end, but without the level of granularity with means everything has it’s own project/repo.

Our current thinking is to create repositories for ‘larger projects’ and to have a Prod & Test branch in them. Individual work streams can be created and linked to an individual project in SL Dev.
Once the development is done, a pull request to merge the artefacts into a Test branch, which can then be sync’d to a single project in SL Test. Test could then merge straight into Prod.

It would mean that Test and Prod may have 30 artefacts in one project but at the moment, this seems to feel nicer than having 10 projects with 2/3 artefacts in them. We could obviously breakdown larger projects through projects within a project space.

Would appreciate any advice people have on their experiences

[akidave]

SnapLogic integration with GitHub is at the project level, a project in SnapLogic maps to a unique repository in GitHub. There is a limit of 200 assets in one version-controlled SnapLogic project, this is in place to avoid performance issues when syncing projects. The projects need to be organized accounting for these two restrictions.

The workflow we have used internally in SnapLogic is

1. Create a project in the development org, checkout the repo from the develop branch (this is done in the SnapLogic Manager). In the production org, checkout the repo from main branch.
2. Make pipeline changes in the development project (in Designer), commit changes into GitHub (from the Manager)
3. Merge develop branch into the main branch based on the production release schedule. This has to be done outside of SnapLogic, directly in GitHub UI or through git CLI.
4. Pull the main branch changes into the production project. This can be done through the Manager or through the public API

Using dynamic accounts, with the env specific values in an expression library allows pipelines with accounts to work across orgs. The accounts can be checked into GitHub or they can be left as untracked assets, in which case the required accounts will have to be created manually in each org.

It is also possible to include peer reviews for project updates. That would require the pipeline developer to create a private branch off develop, checkout a project referencing the private branch and checkin the pipeline changes to the branch. Review requests can be generated between the private branch and develop before the updates are merged into develop branch. The review will show diffs with pretty printed JSON, which works reasonably well. Including pipeline screenshot helps with reviews.

[Supratim]

@akidave Great content. I have one question here-
Lets say one scenario- I have commit one pipeline in Dev and we have CI-CD in place which update/migrate same pipeline in git UAT repo, can we see the same update in snap UAT org without pull the request manually? and do we have any option to automate the same in Snaplogic UAT (which is already pointing to same git UAT repo) org without pulling manually ?

[akidave]

Changes done in git do not automatically reflect in SnapLogic. A pull operation has to be performed in SnapLogic to fetch the changes from git into SnapLogic. The public API should allow this to be integrated into a CI/CD pipeline.

[CookieMonster]

Thanks Ajay,
Sounds like we are on the right sort of track then