What were the underlying reasons or business implications for the need to automate business processes?
We are a fast growing company integrating with the Fortune 1000. One of the largest business problems we currently have is reducing the engineering effort of integrating our new customers, aka Employer Partners. We are evolving our data integration strategy to be more scalable so we can serve all the customers that are waiting to onboard to the newly constructed Guild SnapLogic platform, as well as expanding our ability to integrate with alternative SaaS data sources.
Using our legacy system, this onboarding process takes roughly 3-5 days to complete, and for more complex EP’s, it could take up to 7-10 days. The imperative for Guild is to reduce this heavy lift and onboard Employer Partners in a faster and more efficient manner.
Describe your automation use case(s) and business process(es).
Integrating with Fortune 1000 companies employee data via flat files on an SFTP and direct API integrations.
The process of integrating a new Employer Partner consists of injecting new Employer Partner data into our platform, validating file headers, checking the minimum and maximum number of rows, performing decryption, obfuscating PII data in lower environments, and standardizing individual field formats, as well as, transforming the overall output in a canonical schema form.
Describe how you implemented your automations and processes.
We are using a series of parent and child pipelines to pull down flat files S3 run validations, quarantine bad files, and persist the data against an API endpoint while surfacing data that failed to meet validation. We also enabled a direct API integration using the SAP SuccessFactors snap, solving some complex business logic that enabled us to solve one of our largest customer’s needs.
- Employed an SFTP, API, lambda, and S3 stack. By using these protocols, we are able to receive and consume Employer Partner data.
- Implemented EP configurations. Whenever we onboard a new Employer Partner, we run a PostgreSQL script that outputs a JSON configuration shell file. We make a few updates to this file, which is then used as input into a series of parent and child pipelines. In addition to these configuration files, the pipelines also consume the daily data files from our Employer Partners.
- Using expression libraries. These pipelines highly leverage the use of expression libraries that contain a multitude of functions that not only enable our business to dynamically decrypt Employer Partner files, but they also enable us to retrieve custom field names from the configuration file, which then allows us to map the field values to a canonical schema format.
- Utilizing the REST snap pack. By leveraging the power of the REST snaps, we are able to communicate and pass data to our downstream systems.
- Integrating with S3, Splunk and Slack to quickly identify, quarantine, and report validation errors.
- Employ SAP SuccessFactors snap pack to pull data directly from an EP HRIS system.
What were the business results after executing the strategy?
Enabled a product path (direct API integration) that was technically feasible due to maintenance cost in building it in house.
Automating the EP onboarding process greatly reduced the engineering time it took bring a new customer onboard. This significant time reduction has had a direct impact on our ability to scale with the rapid growth of the business. We were able to reduce this time from 2-3 weeks to less than a week.
The strategy has reduced customer on-boarding from employing 4 engineering squads down to 1.
Who was and how were they involved in building out the solution?
4 FTE at Guild, Kamal & Tony from SnapLogic and PS teams.
Shannon Linn built expression libraries, all of their functions, and designed the architecture. Once the PoC was demonstrated by Shannon, a squad of 4 (the Employer Partner Integrations Team) followed her lead and implemented the process for partners like Chipotle’s, Sunrun, JPMC, Target, Kohl’s, Hilton, and Tyson Foods.
Anything else you would like to add?
Previously an on-boarding took 5 engineering teams 2-3 weeks to onboard a customer. With this infrastructure we’ve reduced that work to a single team in less than a week.
Shannon Linn’s work, specifically within the SnapLogic realm, has been innovative.
SnapLogic - Guild Ingestion Flow.pdf (737.6 KB)
