The Data Lake will get Cloudy in 2017
Written by Chief Enterprise Architect, Ravi Dharnikota. Originally posted in SandHIll Magazine.
In 2016, we saw a lot of interesting discussions around the data lake. In the spirit of reflecting on...
Forum Discussion
So ok, I have a growing collection of data files in a “data lake”. I’ve already created metadata describing where they are, what they contain, who owns them etc. It would be very compelling if I could register these files in snaplogic in the same was as I upload actual “Files” to project space. Then I could build patterns and pipelines that unlock the underlying data in my “data lake” without the end user needing to understand where the lake is, what the naming conventions/security keys/formats/encryption/compressions are etc. I’m very new here, perhaps this concept already exists… (something along the lines of a Metadata Registry / Metadata Repository)
The Data Catalog functionality which was added a few releases back addresses this use case. https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/730006083/Data+Catalog+Service
