Forum Discussion

murphy-brian's avatar
murphy-brian
New Contributor III
3 years ago

ELT Snaps in ELT Pipelines do not process data in Design Mode

We are using the ELT Snaps for pushdown DBaaS transformations. In Design Mode they do not perform DML steps which makes it difficult to build and test. We reported this to product development and they are looking for more organizations to request this feature before putting it on their development roadmap. Are others using these ELT snaps? If you are, how are you all doing development without having to always execute the full pipeline instead of using validation in design mode like all other DML snaps not in the ELT snap pack?
If you would like to see this feature get prioritized, please submit an enhancement request so we can get this in the roadmap.

transform

2 Replies

  • psathyanarayan's avatar
    psathyanarayan
    Employee
    3 years ago

    @murphy-brian ,

    The ELT DML snaps (namely the LOAD, INSERT-SELECT, MERGE INTO, SCD2, EXECUTE) were intentionally designed to not execute in Validation mode (Design mode) for the following reasons:

    1. These snaps can inadvertently corrupt or make the data invalid on production target tables if somebody runs validation accidentally on those pipelines. We want to prevent such mishaps from occurring.
    2. Validation mode (Design mode) can run fast when DML snaps are not writing into the target tables, otherwise it will not only increase the performance overheads, but also increase CDW costs.

    In order to understand your problem better, can you provide some example use cases? This will help us in addressing the problem.

    • murphy-brian's avatar
      murphy-brian
      New Contributor III
      3 years ago

      I understand your 2nd point and that is something a customer’s development team should be considering anyway to ensure the usage costs are managed.

      As for your first point other than the obvious push down behavior why would these snaps behave different than to other DML snaps in other snap packs? Isn’t “Design Mode” just doing a limited run of the pipeline snaps? The only way to validate your pipeline is to run it and if there are any exceptions you need to be an admin in the environment to see what Snaplogic has sent to the DB. We have already discussed this with your product team and they told us this was a large level of effort and needed more customers to request this behavior to put it into roadmap, they never mentioned the corruption issue. We would never validate or go into design mode in a production environment any.