SnapLogic - Integration Nation

Karen · ‎11-19-2021

What were the underlying reasons or business implications for the need to innovate and transform the business? Describe the business and technical challenges the customer was facing.

DCU has grown significantly over the past few years, but their systems and integration had not evolved at the pace needed to support this growth. Because of this, their 40+ applications could not present the “Single Version of the Truth”, restricted the ability to develop a cost-effective Self-Service Reporting capability and restricted needed Data Governance capabilities.

The legacy environment also had a lack of automation needed to support the scale desired. The architecture was not capable of supporting the data and process scale required to sustain the goals. Lastly, there was extensive manual data manipulation being performed by multiple departments often leading to disparate results.

The customer use cases were:
• Predictive Analytics - liquidity
• Near real-time reporting focused on fraud
• Proactive collections and member stratification,
• Trend analysis leveraging accurate models

The challenges including un-usable data due to the silos and disconnected systems. THe customer struggled to integrate data or compare across source systems. They also experienced slow, inconsistent, underwriting decisions hampered by data constraints.

Which customer did you implement this on?

Digital Federal Credit Union

Describe your strategy and execution to solve the business challenge. Include details on how SnapLogic played a role in the strategy and execution, including Snaps and other SnapLogic products/features.

EXL Service was engaged to support the goals of DCU: to continue to grow membership as well as grow member products, services & offerings to improve member value. To achieve these goals, we began a multiple phase business and technical analysis of the current environment & operations at DCU. These efforts focused on the following workstreams:

• Deposits and bank statements
• Credit Card transactions
• Early risk predictions
• Attrition models

The project was started with two initial use cases
• Executive Dashboard - focused on monitoring key metrics regarding member activity, deposit analysis, product growth, and key success factors
• Transaction Data Mart - integrated account transaction history providing member behaviour analysis, with identification of product and service opportunities

The management roadmap goals included:
• Improve ability to deliver additional Self-Service Capability
• Focus on Data Quality, Data Definition and Data Ownership
• Need for more integrated data to provide better insights (based on historical data)
• Goal is to enable a Data Driven Culture

From these findings, we proposed a significant modernization effort to address the issues identified. This effort included the design and development of a Cloud-based Modern Data Platform. As DCU was an existing AWS customer, we developed the solution on AWS using both Amazon and partner technologies. This platform will address the current issues, plus provide the sustainable, scalable technology base needed for the next phase of growth.

Together with SnapLogic we proposed an archtiecture that leveraged SnapLogic with Amazon Redshift for the core capabilities. SnapLogic was able to demonstrate their maturity, performance and features that drove the automation, scale and managed processes needed. SnapLogic’s integration with AWS IAM and Security were critical to building a compliant solution in a highly regulated environment.

The DCU effort is one of the first solutions to integrate SnapLogic with AWS Fargate - a serverless architecture that leverages the AWS Elastic Container Service. In addition to the rapid scalability, SnapLogics integration with multiple AWS services including Secrets Manager, Key Management Services (KMS) and encryption in the S3 Data Lake is unique. SnapLogic allows us to split out columns based upon sensitivity and place those columns in different RedShift schemas to enable appropriate user access to data with different levels of privacy (non-sensitive, sensitive and secure).

SnapLogic is heavily leveraged for data flows between systems (DB2/SQL Server/MySQL → Parquet files in S3 → Redshift). For operations within Redshift, we have SnapLogic generate dynamic DML based on metadata and “push down” data transformations to Redshift.

Using SnapLogic tasks that can be executed via URL calls, we leveraged an on-prem enterprise scheduler that allowed DCU to link internal dependencies to when SnapLogic tasks should be launched, and the timing was different and advanced taking into account banking holidays, as well as end-of-month timing differences. As opposed to just building API’s with SnapLogic, we are using SnapLogic as an ELT / ETL tool to build out a full blown modern data platform in the cloud, including a Data Lake in S3, Parquet tables, and well-architected data models in Redshift and Aurora to support advanced BI tools like Tableau and soon to come advanced data analytics.

The dynamic approach taken with the pipelines means that we only need to develop a set of templates based upon what the source system database engine is. SnapLogic’s ability to generate code “on the fly”, allows pulling in metadata and acquiring new tables, without any additional development necessary. SnapLogic’s ability to create complex execution paths within the pipelines allows for conditional execution of pipelines depending upon the nature of the source table (is it a type 2 or a type 1 or an append of transactional data).

Who was and how were they involved in building out the solution? (Please include the # of FTEs, any partners or SnapLogic professional services who were involved on the implementation)

The project currently has 23 FTE working on this solution. We have worked with SnapLogic in the integration with AWS native services and specific performance tuning of the PII data being protected. In particular we have engaged with the SnapLogic customer success team, which consisted of Praneal (sales) and Roger (engineer.)

What were the business results after executing the strategy? Describe how your company, departments, and/or employees/customers benefit from the application integrations. Include any measurable metrics.

The project is still in process with an anticipated completion within the next few months. To date, the project architecture, AWS Services and SnapLogic automation has achieved the following:

• A well-integrated modern data platform in the cloud leveraging on-prem source systems, several AWS services (FarGate, RedShift, Aurora, S3, Secrets Manager) with, currently, an executive dashboard (Tableau) on top of well-architected data model.
• Executive acceptance of the data being presented, with their request to leverage the dashboard in lieu of legacy reports for all of their reporting needs by next year.
• Initiation of projects to retire legacy on prem reporting databases, which will save DCU significant money in maintenance and human resources
• Projects being initiated to start advanced analytics and data science efforts.

What was the ROI the customer gained from executing the strategy? Include any additional measurable metrics (ie. productivity improvement, speed improvement, % reduced manual inefficiencies, etc.)

The solution is still in construction, but over the next 1 to 2 years, we anticipate achieving the following goals:
• 80% reduction in manual processes
• 20% reduction in duplicated data across departments
• Scalable solution to address 100% of performance constraints
• 30% reduction in data quality issues and corresponding costs and customer issues.

These transformations are supporting the customer objectives of well-managed data (read: proper integration solutions) leading to consistent metrics, and an ever-learning, high achieving organization.

Early commentary from the senior leadership team has been extremely positive regarding the teaming of EXL and SnapLogic. They have been impressed with the level of rigor we bring with our solutions, the speed with which we can deploy and the quality and learning we are delivering together.

Anything else you would like to add?

Through proper integration (i.e. using our Data Model Accelerator plus SnapLogic solution) combined with solid business rules we aim to ensure quality analytics through sound architecture and best practices (across DG, DQ Mgmt and Data Mgmt).

The foundational impact of EXL’s implementation of our Accelerator and SnapLogic technology is changing the game for DCU with 2 key themes “speed & flexibility”. The successful teaming of EXL and SnapLogic is already producing innovation for both partners as well as achieving the customer goals and transforming their technology platform, building a data-driven culture and positioning the organization for sustainable success. We believe that this could not have been achieved without the cohesive teamwork enabled through the partnership.