vsunilbabu
New Contributor II
Joined 7 years ago
User Profile
User Widgets
Contributions
Re: Change datatypes dynamically of every column
Hello, Can anyone look into this please. Is this an issue with Snowflake snaps? I used Snowflake Bulk Load snap many times before to upload data. Usually, I create necessary tables using DDL statements before hand in snowflake and then use Bulk Load snap in my pipelines, but this is the first time am using ‘Create table if not present’. Thanks, Sunil1.3KViews0likes0CommentsRe: Change datatypes dynamically of every column
Can I ask a question on top of this. I output of my mapper has correct datatypes, I placed a Snowflake Bulk Load snap after the mapper with below configuration This snap should create a table in the database if it is not already existing. Pipeline ran successfully and the table was also created. But when I see the datatypes in my database, all the columns are Varchar Is there any specific reason for this. How can I ensure that the datatypes at the end of mapper to reflect in my database(snowflake)? Thanks, Sunil3.4KViews0likes1CommentRe: Change datatypes dynamically of every column
Because I was stuck, unable to come up with list of column names to pass into pipeline execute as parameters, I used CSV generator initially to test my logic. I could not do it. Any help is appreciated. Pipeline should look at the first row and change the datatype of that column. Thanks in advance. Regards, Sunil3.4KViews0likes4CommentsChange datatypes dynamically of every column
Hello everybody, Need help creating a pipeline. The data am working on looks like below: Header Datatype Data Data Data This is a CSV file so every column comes as a string. I want to create a pipeline which will look at the first row and change the datatype of that particular column using mapper. My thought was to initially find only columns in the form of a table(group) and pass one by one as a parameter into a mapper placed within a pipeline execute. The mapper within the pipeline will have something similar to group[1].contains(‘date’)?Date.parse(_parameter):group[1].contains(‘char’)?_parameter:group[1].contains(‘char’)?_parameter:parseInt(_parameter) Can anyone help me how to bring out only column names as an array/table so that I can try out my logic. (OR) if you have any other easier method to do this, then can you please share. Thanks, Sunil6.9KViews0likes6CommentsSeparate words in column name
Hello, Can you please help separating my column names using ‘_’ whenever there are capital letters in the column name. For example: Input Output In my research, I understood that findall() function can help iterating through all column names, using the logic: output.append(“_”.join(re.findall(‘[A-Z][^A-Z]*’, i))) Can someone help me with a script for this? Or if there is any other better way. Thanks, Sunil2.3KViews0likes2Comments