09-29-2023 08:00 AM
I Using saleforce read snap to read data and write to s3 in parquet format but getting error due to 'picklist' data type which is not compatible to parquet. How to handle such datat types problem
Solved! Go to Solution.
10-04-2023 01:56 AM
Hi @Roger667
Could you please check your SF read snap and make sure the 'Match datatype' is checked.
Thanks !
10-04-2023 09:49 PM
Hi @Roger667
I'm not sure of this behavior, I haven't faced this earlier and not able understand the issue. may be you can try copying the contents of the file in a new CSV and use it.
mapping CSV file is an option to convert the metadata, you can use an expression file also to achieve this. I found this easier and used this. but you can also maintain these details in an expression file and call it in a mapper to get the parquet appropriate data types.
Thanks !
10-06-2023 02:50 AM - edited 10-06-2023 03:38 AM
Hi @manichandana_ch
This error is comimg due to datetime data type in salesforce object which is mapped as bigint in the mapping sheet. But this mapping leads to loss of data and timestamp only comes as year. Can you suggest how to handle this datetime
10-08-2023 09:45 PM
Hi @Roger667 ,
timestamp is converted to bigint as parquet writer didn't work very well when we tried writing timestamp, hence we converted to bigint and it converts to unix time format. then we cast the timestamp columns when we/downstreams read the file. if bigint didn't work for you, you can use string format and then cast it while reading the file.
Thanks !