Change datatypes dynamically of every column

[vsunilbabu]

Hello everybody,

Need help creating a pipeline. The data am working on looks like below:

Header
Datatype
Data
Data
Data

This is a CSV file so every column comes as a string. I want to create a pipeline which will look at the first row and change the datatype of that particular column using mapper.

My thought was to initially find only columns in the form of a table(group) and pass one by one as a parameter into a mapper placed within a pipeline execute. The mapper within the pipeline will have something similar to group[1].contains(‘date’)?Date.parse(_parameter):group[1].contains(‘char’)?_parameter:group[1].contains(‘char’)?_parameter:parseInt(_parameter)

Can anyone help me how to bring out only column names as an array/table so that I can try out my logic.

(OR) if you have any other easier method to do this, then can you please share.

Thanks,
Sunil

[vsunilbabu]

Hello,

Can anyone look into this please. Is this an issue with Snowflake snaps?

I used Snowflake Bulk Load snap many times before to upload data. Usually, I create necessary tables using DDL statements before hand in snowflake and then use Bulk Load snap in my pipelines, but this is the first time am using ‘Create table if not present’.

Thanks,
Sunil

[smanoharan]

@vsunilbabu If Create new table if not present option is selected without providing the schema in a secondary input view, varchar will be used for all the column’s data types.

In your use case, you can provide the schema of the table that you want to create in a second input view to get exactly the data type that you want in the Snowflake Table.

The first example mentioned in the Snowflake Bulk Load snap’s documentation covers a similar use case with an example: https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438549/Snowflake+-+Bulk+Load

cc: @dmiller