Real-World Insights on Successful Deployment of Integration Across Enterprises

[Dominic]

For this episode, @GuyM and I invited @chrisward to join us to talk about best practices for successful adoption at enterprise scale, how to establish a Center of Excellence, and what are some common pitfalls to avoid. Chris is a Senior Solution Architect, working on ensuring the success of some of SnapLogic's largest and most strategic customer deployments.

Chapters

• Best Practices for Enterprise Architect Teams: the Sigma Framework
• Platform Adoption and Measurement Strategies
• Continuing Conversations With Enterprise Alchemists

Transcript

Dominic Wellington: 0:00

Hello and welcome back to the Enterprise Alchemists. I'm Dominic Wellington and I'm here with my colleague, uy Murphy from SnapLogic. Today we have a fantastic guest. We're talking to Chris Ward, who is one of our top architects on the services side. Chris, why don't you introduce yourself briefly?

Chris Ward: 0:18

Absolutely yeah. So yeah, my name is Chris Ward. I'm a Senior Solutions Architect within the Professional Services team here at SnapLogic. My focus day-to-day as an architect is really to ensure that our customers adopt the platform in the right way, obviously aligning with architectural best practices and focusing not just on for individual project use cases, but also sort of more broadly to help ensure customers can scale effectively, really by helping them develop cross-platform capabilities as the platform is adopted at scale across the enterprise.

Dominic Wellington: 0:54

Thanks, Chris, and welcome to the show. And of course, Chris's activities are one of the major sources for what we on the Enterprise Architect team then try to codify into best practices as part of the Sigma framework. Guy, why don't you give a brief intro to what Sigma is all about?

Guy Murphy: 1:14

Yeah, absolutely, Dominic, and again thanks for joining us, Chris.

Guy Murphy: 1:18

So for several people in SnapLogic, they're probably aware that Sigma has been near and dear to my heart for about the last two and a half years here.

Guy Murphy: 1:26

So the Sigma framework is really a concept of bringing together best practices and also linking it to the types of decision making processes that mainly medium and large enterprises really need to think about when they're looking to roll out a technology like SnapLogic, and obviously we then combine it with more detailed macro best practices of our platform. The aim is really to understand what the customer's strategy is, what they're trying to get out of the platform, how they see success and then, moving into other concepts, so how they need to think about structuring a team, what the culture of the team is going to be everything from centralized traditional IT right through to incredibly federated citizen developer type solutions and one of the interesting things with SnapLogic is the sheer breadth of capability. But that also then causes us to have some interesting challenges, because our breadth of best practices is. That also then causes us to have some interesting challenges, because our breadth of best practices is a lot wider than most point technologies in the marketplace.

Dominic Wellington: 2:31

Exactly, and a lot of customers will have very similar requirements, although, you know, at the top end of the size and complexity scale there's a lot of custom work, inevitably as well.

Dominic Wellington: 2:48

But there is a solid foundation that is going to be pretty similar from one deployment to another, and that's what we want to make it easier to achieve to get that foundation dealt with so you can focus on the custom stuff that's actually differentiating. And one of the big things that come out of that work, that analysis of successful deployments, is that recommendation to set up a center of excellence, to have a single point of contact that people know to go to in order to get things done in the platform and do them in the right way, and to think strategically about what the overall platform should be doing, and so on. And so, Chris, having set up a few of these for various of our customers, why don't you talk about what is needed to make that work, to make sure that the deployments are successful?

Chris Ward: 3:35

Absolutely yeah. So, yeah, there's quite a lot of different things we can start to think about. To think about, I'd say, first and foremost, one of the biggest things we can do to help COEs start to evangelize across the developer community, within their organization, is to really start thinking about the concept of reusability, particularly in the integration space. So one thing you touched on slightly there, Dominic, was the fact that obviously different project teams, different business domains are building integrations within their own independent constituencies.

Guy Murphy: 4:18

So Chris, obviously we've been working across a number of accounts around EMEA and Asia and North America. One of the things that's been really clear and we actually touched upon it in our first podcast is what's the concept of modern architecture and how does it integrate into the organization of a company? And people have been talking about the concept of a COE, a center of excellence, or a center for enablement is a more modern version of it, and really it seems to be this sort of tension and want to move from a 100% centralized IT organization to a highly federated environment, often driven by concepts like agile development and others. What are you seeing across the landscape and when they you see customers wanting to adopt SnapL ogic, what's the type of things you advise them on making decisions about the right type of team and what sort of things do they need to think about if they're going to support these different types of team structures?

Chris Ward: 5:23

So if we look at sort of how teams operate and go about their day to day work, I think there's certainly a need to build a broad set of skill sets, more sort of foundational skill sets.

Chris Ward: 5:42

Foundational skill sets, so really looking at starting thinking about some of the core concepts to deployment, testability of integrations, examples like CI/CD, obviously a big facilitator of that as well and then some more on the technical side of things, so thinking about patterns and how teams can build sort of composability around what they're aiming to deliver. It's on the non-technical side of things, then. So it's really sort of operational, day-to-day how the teams work with one another, operational, day-to-day how the teams work with one another, and I think one thing that teams can really start to focus on to help improve the way they work is to sort of share amongst themselves the types of problems and challenges that they're working on, different levels of complexity obviously, and what that really helps do is build out a less of a tribal sort of knowledge around certain business areas. So it's really looking to sort of share. What they're doing is their opportunity to to build something in a way that that is more composable and reusable, not just within their own project team but more broadly, as well.

Guy Murphy: 7:06

So that's obviously quite a different cadence from the sort of traditional centralized teams. So again, being quite open, if you had a team of 20 or 30 developers under a couple of project managers operating a technology, knowledge sharing was quote-unquote quite simple, though the reality was often too many Word docs, too many Visio diagrams, too many Excel spreadsheets being the backbone of an architecture.

Dominic Wellington: 7:33

And that works as long as you have a small team working physically close to each other and everyone can shout at each other or you can catch up over coffee, but it breaks down when you have a bigger, distributed team.

Guy Murphy: 7:44

Absolutely so. And again, we've got clients that have got hundreds, in some cases even thousands, of citizen developers. So could you touch upon, if you're trying to own a platform, what type of things do you need to do to support this much more distributed team? So communications, comms, portals, well you know, what's the infrastructure for supporting a federated world.

Guy Murphy: 8:10

Where in the central world it was easier to say, you know, put it all on the shared drive, put it in the shared environment, put artifacts in git, move on. We've both seen that this is actually quite a challenge for traditional IT orgs to migrate to this sort of different way of thinking about a team.

Chris Ward: 8:27

Absolutely yeah.

Chris Ward: 8:28

So that's a great question.

Chris Ward: 8:29

So I think, really, if we were to look at a way to easily enable teams at scale it's thinking about building self-service capabilities within the organization that allow you know these hundreds of thousands of developers that are building integrations in the platform to not have to go to an individual person and ask for something, but rather being able to log into a platform.

Chris Ward: 8:59

That is really a guided, assisted way of requesting something, be it a credential, into a database or an application, and also as well, maybe triggering a deployment activity or maybe onboarding users within the platform, performing architectural reviews on what the teams have built. So really looking at sort of automating some of those manual steps that you would typically see an individual conducting. Aside from that as well, I think it's really important to think about having that cross communication that I discussed a second ago, which is having regular open office hour sessions with the developer community, run by the central COE team or maybe a project team as well. Maybe there's something that somebody has built that is really worth sharing, maybe a success story of how they approached and delivered a complex integration problem, and it's really looking to collaborate, share knowledge through those different channels to drive success.

Guy Murphy: 10:15

You bring up a couple of really interesting points there. One of them for our listeners. I'd strongly advocate thinking about this. So, as a platform matures up, you'll end up having a series of interlinked best practices patterns, application patterns, etl patterns, file movement patterns, maybe more granular ones down to technology stacks say, best practices around operating with Snowflake but you will come across teams that will then go outside of the standard scope.

Dominic Wellington: 10:43

No, surely not!

Guy Murphy: 10:46

Indeed. No, in a good way.

Guy Murphy: 10:49

Business is always moving forward and traditionally there's been a perception that the central IT teams, the COEs, actually act as a hindrance on bringing in new patterns and new capabilities.

Guy Murphy: 11:01

I'd actually advocate that teams embrace these new use cases and actually put more focus into them. So SnapLogic has a large footprint in manufacturing and over the last 12 months I've personally seen the rise of event-driven integrations into factory automation really starting to roll out as a much higher level, using much more generic technologies than traditional. And again, rather than than blocking this, what we're seeing is our customers are actually embracing it and saying what? How do we embed these types of new patterns into the snap logic environment? How do we test and scale it when it's a completely different concept? Maybe they've done before and then how do we codify that so that the next project along will actually get a better, faster rollout of these sorts of patterns? So it's sometimes easy to believe that the CIOs actually become a non-sequitur in these. But actually having a centre of excellence team that can actually act as a switch and an investment team and preferably be also technical gurus as well, I see time and time again to pay dividends to enterprises.

Dominic Wellington: 12:11

The role of the COE should not be to be the bottleneck through which everything must pass, but the enabler to set things up so that individual teams can go and do their own thing, but give them the tools and the structures and the processes so that what they do is done in a way that will pay dividends in the future and not become a technical debt that then has to be dealt with at some point by someone with a shovel.

Guy Murphy: 12:37

Chris, you made a provocative comment in your previous statement, which was developers being able to not just self-service and self-enable but even take integrations into production, and, being an old, crusty, skeptical architect that I am, that could put a raise hairs on the back of my neck about the idea of people randomly pushing code into production without any oversight.

Guy Murphy: 13:10

Now I know that that's not the premise for what your comment was, but it is an interesting balancing act. The dream of citizen developer is total ownership of process, and yet we also have many scenarios where our platform is being used for true mission critical break the business type, processes being application or data. How do you see balancing out these two different concepts in the accounts that you're working with?

Chris Ward: 13:35

Great question, yeah, so I think it's really twofold.

Chris Ward: 13:38

So, firstly, it's really important for the centralized team to help educate the developers on the platform of how they do things the right way. First and foremost, when they approach a new problem, they build an integration pipeline, one of the things that I need to start thinking about: n that may or we may in, of build risk mitigation auditing, error, for example, auditing um really minimize or uh eliminate any risk uh down the line when the integration runs in a production setting. So it's having Dominic, as you mentioned, dominic, the ability to empower and educate the teams to do the right

Chris Ward: 14:31

thing from the outset. And then, second to that as well, is when the teams have built what they've built and they want to go through the cycle of deployment, moving through the different environments, is really to conduct or build automation around reviewing what has been built in a way that again doesn't really require somebody to visually look at what's being built, but rather trigger some automation that's going to do an architecture review, highlight any areas that look concerning or may again introduce risk down the line, and then folding that into a broader CICD lifecycle so that developers can push their pipelines, their code. It triggers effectively the architecture review in an automated way maybe provides feedback so the developers can iterate and improve what they've done based on

Chris Ward: 15:35

that feedback going forward.

Chris Ward: 15:38

. Yeah, at a foundational level surface any areas done, by for what has been it; effectively.

Guy Murphy: 15:49

Chris, this sounds like a lot needs to be done. By the sounds of it, it might sound a little bit daunting. Obviously, we're not in the age of the 90s or noughties where we can afford to stop the world and build out a platform for three years before delivering any value. How do you see your team and SnapLogic obviously working with accounts to start this journey, but actually achieve things quickly while they're building things out?

Chris Ward: 16:18

Yeah, sure.

Chris Ward: 16:19

So it's been really interesting to see, across the range of accounts that we have, customers work with day to day, the varying level of maturity and existing sort of skill sets that they have within their organization. So the first thing level we want A try and understand is where those customers are at in that level of maturity. Are they low, are they medium, are they high? Have they adopted advanced to do really clever things? And then sort of between. That is, once we have that understanding, really looking to see the customer get to, you know, over the next one, two, three and how we support that

Guy Murphy: 17:05

.

Chris Ward: 17:06

And I think a key thing for them to start thinking about is really, are their teams sort of geared and set up for success? So thinking about some of the foundational skill sets that are needed to help with that journey at a foundational level. So a lot of cloud providers now provide easy ways to upskill in certain areas, things like DevOps, managing the infrastructure, observability. So I think it's really important for for customers to sort of understand where, where that team is at um and how we can help them get to that that level um, and then, yeah, addressing any gaps, highlighting the gaps in in those areas and um assisting with those sort of upscaling activities and for many listeners, um also being quite candid and open, we also see the other extreme of the maturity curves.

Guy Murphy: 18:02

And for many listeners, also being quite candid and open, we also see the other extreme of the maturity curve every now and again. I might have been sounding sarcastic earlier, but we have come across customers who were initially running our platform completely manually and using spreadsheets and Word documents as the backbone of it. So it can be done…

Dominic Wellington: 18:23

Wouldn't recommend that!

Guy Murphy: 18:25

Not recommended, no and, more importantly, it becomes in the medium to long term a cost, being the habit of rolling a platform like SnapL ogic out at scale. So if you were sitting there thinking "I haven't got all this modern automation, don't panic. You're probably not always as alone as you may think you are.

Dominic Wellington: 18:45

Yeah, and I think that's a very important point, because the users of the SnapLogic platform, especially if you're going all in on the citizen developer mode, they're not developers. They are people who come from the line of business. They might not have the sort of reflex that a more traditional developer would have that of course there will be version control. Of course there will be a release process to go into production, etc. etc. Now the platform of course has all these hooks. It is, for all intents and purposes, a modern development stack, but that's why the COE is required to make sure that people who don't have that sort of just muscle memory and reflex and expectation and know that this is something that they should be doing and have it all set up so that they can do it easily and that it doesn't become a drag on their business velocity.

Guy Murphy: 19:37

Absolutely. Chris, you mentioned something kind of interesting there which was like the measurement of the platform, and we've talked quite extensively about this over the last couple of years. I've had the pleasure to work with you. Can you touch upon why is measurement so critical in the platform? And obviously I'm not just talking about the technical capabilities but trying to align it with business outcomes when you look at something like SnapLogic, because some cynics and skeptics out there might say it's just another integration platform, but obviously we believe that. The fact is we are the glue between mission-critical systems. We are the glue between fast moving projects and in many, many of our accounts, we are the platform that is actually core process. What's your experience about the different levels of maturity of measurement and then how customers move from simple standard measurement of a platform through to much more sophisticated type of use cases, and I'm thinking about some of the more advanced dashboarding that you've done for some of our larger customers.

Chris Ward: 20:51

Yeah, great question, I think, sort of going back to the maturity discussion we had.

Chris Ward: 20:58

It's really looking at building measurable KPIs metrics around, how customers can help to understand how they're tracking on that maturity curve, and one thing or a few different things we've done with various customers is to help them understand the different metrics and data points that are available within the platform specifically that they can draw upon to help build that KPI framework dashboarding and insights around.

Chris Ward: 21:30

And what it really does is it helps not just measure ongoing, but it helps facilitate conversation, I think, among the different business units on a technical and a business level, to see, from an adoption point of view, where they're tracking, if you may have a marketing function or a sales function that are building integrations in their own domain and they may not know month on month how much data they're using, they're consuming, the frequency of what they're building as well. So the dashboarding and the insights really help surface that and have a conversation around. Well, is there additional capacity maybe required to support from an infrastructure point of view with what you're trying to do? And again, it really helps central teams understand on the domain level, the business level, things like consumption patterns and even help with maybe understanding costs and chargeback models as well. So, yeah, it's building that over time. I think it's important to maybe see it as a gradual thing, having some base level KPIs that can be used from the outset.

Dominic Wellington: 22:55

And this sort of benchmarking against other customers to know how well you're doing, is there anything else that you're supposed to be doing at this point in your maturity, is exactly what the Sigma framework is supposed to be about, and so, for instance, Chris and I collaborated on a document about shared development best practices. I'll put the link to that in the show notes for those who want to read it. And, of course, all of these materials are on the Sigmawork library on SnapLogic's Integration Nation community website

Dominic Wellington: 23:29

but the Sigma Framework always comes from a real world experience. All of these things that were developed in the field by Chris and other members of his team, but Guy and I wanted to get Chris on the call because we both worked together on some of SnapLogic's largest and most strategic deployments. So could you share maybe a couple of stories based on things that you've seen in the field that might give us some indications of what to do and perhaps, even more importantly, what not to do, when getting started on setting up a platform that you want to achieve? absolutely

Dominic Wellington: 24:06

yeah. . .

Chris Ward: 24:06

So I'll start with the. What not to do, the really the. I guess the pitfalls uh to to avoid, or I'd recommend um thinking about, is, uh, teams, I think, have a tendency to maybe operate in a siloed way. Again, going back to the whole community discussion, we want to help fix that through, you know, broader collaboration and communication. It was in the enterprise um is yeah, if you've got a very domain focused team, um, I think what, what can happen there is you get that whole tribal knowledge thing going on and there's also, I see, quite the tendency to over-engineer solutions as well. So it's really helping to step back, look at the bigger picture and building, I guess, a culture of leading with an iterative sort of mindset, building MVPs, proof of concepts out just to explore a potential approach to an integration problem.

Guy Murphy: 25:11

Could you quickly give a bit more of a concrete description of the over-engineering, because I think that's a very interesting and important concept for us to talk about.

Chris Ward: 25:24

So the over-engineering exists in any um, any sort of technical domain, I guess. I think that, particularly with non-technical users that, again, are using our platform or other platforms in a citizen way, is they don't come with the background of a developer that may look to approach a problem in a an abstract way, and I think that what can easily happen is you have all these really cool tools at your disposal that you can start dragging and dropping, connecting together, and I think sometimes with that ease of use it is almost easy to forget, what is the problem that I'm trying to solve here? And that leads to maybe doing something in a way that isn't optimal, maybe gets you to the end goal, but it almost becomes a bit of a behemoth over time and very hard for other users of the platform or people picking up that integration maybe later down the line to decipher what has happened. And there's also risks that come with overengineering things from an infrastructure point of view, it may lead to spikes in memory consumption, potentially as well.

Guy Murphy: 26:56

So that's really interesting. So I guess you know where we focus then is, as you said earlier, reusable artifacts, things like the patterns library. Also within the platform there's the SnapGPT-enabled admin capabilities that allow you to profile pipelines where you might actually be able to see overly verbose capabilities or, as you said, incorrect or maybe not optimal patterns in. It is an automatic feature and, again, I know this is something we've introduced to several accounts around the onboarding process and that will help mitigate it, but it's an interesting point to consider because it's almost counterintuitive. The ease of use can actually lead to complexity because it's almost just too simple to throw together what you think is rich capability and actually sometimes getting simplistic if you talk to any philosopher is actually harder to do in the on-term. So no, that's really an interesting dynamic. You're seeing what other are the top two or three sins that you've seen or challenges when trying to work around rolling out a platform into a new account?

Chris Ward: 28:05

Yeah. So one of the other things I see occur quite frequently is the tendency to avoid or not avoid, maybe not see the need to define measurable business outcomes from the start and almost just dive in and start building an integration. So really that may get you to having something up and running relatively quickly. But what it is quite hard to do going forward, Dan, is measure the effectiveness of what you've built over time. So, having concrete business outcomes that maybe NFRs or other domain business level, what is this integration actually going to do for the business? Setting those out and then, going back to the KPI discussion, you can have measurable metrics, measurements around how that's performing relative to those business outcomes.

Guy Murphy: 29:13

Chris, again, 100% in agreement, and I'm actually going to call this out in a more stark manner. We've actually got a couple of accounts that have paradoxically, been incredibly successful in rolling us out, but then we've had new senior management come into the company and asked very, very logical questions what did the platform do to us? What was it we were trying to achieve? And the paradox is we can tell the customer how fast and flexible we are and how much data we're moving, but actually no data was actually captured on the original, as is so the upshot ways is. Actually articulating that when a platform is successful is very hard, because often the bad old days is quickly forgotten. So, again, this podcast is probably more aimed at architects and enterprise architects.

Guy Murphy: 30:08

I would strongly recommend, if you're adopting a platform like SnapLogic, is document what it was like beforehand, document how long it took to create developments. Document quality issues, document long-term costs of point-to-point integrations. Have that as a sub-business case, because I will happily say, in a very long career in the integration domain, somebody at some point is going to ask the question what did you do for me? What was the investment and what was the return? And I'm still surprised, in the 2020s, how many customers actually struggle with truly articulating that we were very fixated on the today, but that inability to do a compare and contrast can be quite painful. If you haven't done that piece of work, so write the doc, grab the spreadsheet, get the data, put it on the drive, park it. You will thank us when you come back in two or three years' time when the new CFO or CTO comes on board and says what was the returns on that investment?

Chris Ward: 31:17

Yeah, just to add to that as well, I think that's a great point. So if I was thinking of adopting an integration platform, a new integration platform to bring into the business and I have the luxury to start defining what good looks like, I would do it from the start. I think some of the challenges that I've seen, particularly with the larger enterprises, is retrofitting that way of thinking that you just talked about, guy, into something that has already been adopted at scale. So, yeah, absolutely Think about it from the start.

Dominic Wellington: 31:55

Vigorous agreement from Guy and me. This is an audio-only recording, but the three of us can see each other and there's some neck-snapping, nodding going on. But yeah, thanks, chris. That's a wonderful point to end on. I think so.

Dominic Wellington: 32:12

If we distill everything that we've said down, I think the main point is the CoE is there to empower teams to do the right thing, to help them implement and move as fast as they need to to satisfy their business requirements.

Dominic Wellington: 32:27

But with that very important sub- point: make sure that you understand their business requirements and document them so that you can track how they change, measure the vector of improvement, take any steps that are needed to ensure that you're getting maximum value from the investment in time and resources and training, et cetera, et cetera, that goes around adopting a platform like SnapLogic.

Dominic Wellington: 32:52

So thank you very much, Chris, for joining us, for keeping it real with your war stories, always a pleasure to chat and we'll be sure to have you back on, because in the planning for this episode we realized we'd come up with enough topics for at least two and a half conversations like this one, so we've got plenty of material already for season two and even season three. I'm sure we'll be talking to you again soon!

Chris Ward: 33:18

A bsolutely, happy to come back and keep the conversation going!

Dominic Wellington: 33:25

As ever, thank you for tuning in to the Enterprise Alchemists. We hope you found this conversation interesting. If you want to join in, we have companion blog posts for each episode that go up on the Integration Nation website. Links will be in the show notes for this episode, and those blog posts also feature full episode transcripts for you to go back and read through or for you to share with any colleagues who might prefer the written medium, as well as links to any resources that we mentioned during the course of the show. But with that, thank you once again and we'll be talking to you again next week.