Tuomas
New Contributor II
Joined 3 years ago
User Profile
User Widgets
Contributions
Re: Handling missing object properties with Mapper
Hi koryknick ! Thanks for response! Yes, I have the Null-safe access enabled. In this case it doesn't work though. I have "labels" as my output field, not "fi_FI" and "en_US". Thus, when "labels" in input is completely missing, then I will have null output for "labels" but also I'm missing both the properties. Sure, one way to do it would be to map each property to its own output field which I have done in the past when facing the similar problem. But I figured maybe I should find an alternative solution as I don't really need the "labels"-object to be split. Also, if I hard code the "labels"-object, I might lose something in the future if more properties are suddenly added to it. Essentially, the missing properties were problematic when loading data to Big Query. An object with one of the properties missing happened to be the first object loaded which the connector apparently uses to determine Big Query table's schema. Big Query adds objects to table as "labels.fi_FI", "labels.en_US", therefore a property missing from first object created the table without that column and breaking the load for subsequent objects. Hopefully this clarified the issue 🙂3.4KViews0likes0CommentsHandling missing object properties with Mapper
Hello, I've been struggling with this for few hours now and I simply can't get it to work as I want to. Therefore I'm asking for help. So I have the following input incoming to Mapper: { "id": 1, "labels": { "fi_FI": "label_1" } }, { "id":2, "labels": { "fi_FI": "label_2", "en_US": "label_3" } } So I have a missing en_US label on the first object. How do I add a null or blank en_US label? So the output would look like: { "id": 1, "labels": { "fi_FI": "label_1", "en_US": null } }, { "id":2, "labels": { "fi_FI": "label_2", "en_US": "label_3" } } There can be possibly be X more labels-properties. Fully dynamic solution would be the best, but I could work with determining all the labels-properties as well. Cheers!3.5KViews0likes3CommentsMapper dynamic content
Hello! I have REST API source from which I ingest data and load it to BigQuery. I am executing this dynamically with parameters as there are multiple endpoints to query. I am using mapper to define the schema before loading, in this case with the pass-through enabled. However, I have run into a problem, that sometimes the API doesn’t return certain object in the json. Then, when I auto-create the table during load, it may not define this missing column in the schema based on the sample, but then breaks when this column suddenly appears. Due to this being a dynamic pipeline, I cannot hard-code any single columns to the mapper, as the column names are different for each job. Also, the number of columns might differ. So is there a way to dynamically define the column names I want to keep from the source? For example I a good solution could be one where I can define a single json, array or some other sort of string that contains the namespace of columns I’d like to keep, and just give that as a parameter to the mapper. Just to showcase with an example what I would want out with the dynamic expression: Source 1: [ { “a”:“123”, “b”:“qwerty”, “c”:“aaa”, “d”:asd", “e”:“111” }, { “a”:“456”, “b”:“zxcvbn”, “c”:“bbb”, “d”:fgh", “e”:“222” } ] Target schema 1: a, b, c, e Source 2: { “h”:“123”, “i”:“qwerty”, “j”:“aaa”, “k”:asd", “l”:“111” }, Target schema 2: i, j, k Hopefully you get what I’m after 🙂 Thanks -TuomasSolved5.9KViews0likes5Comments