Forum Discussion

Roger667's avatar
Roger667
New Contributor III
3 years ago
Solved

Unable to write salesforce data to s3 in parquet format

I Using saleforce read snap to read data and write to s3 in parquet format but getting error due to 'picklist' data type which is not compatible to parquet. How to handle such datat types problem
  • manichandana_ch's avatar
    manichandana_ch
    3 years ago

    Hi Roger667 

    Could you please check your SF read snap and make sure the 'Match datatype' is checked.

    Thanks !

Roger667's avatar
Roger667
New Contributor III
3 years ago

Thank manichandana_ch . This solution worked for me. I have a question that may extend beyond the current topic. How should I design the pipeline to accommodate incremental loads?

  • manichandana_ch's avatar
    manichandana_ch
    New Contributor III
    3 years ago

    Hi Roger667 

    After getting the data from SF read snap, you can probably take a copy and get the max value of lastmodified date or any audit date column and store it in a file or config table if any. Then, before the SF read, you can read the file/table and get the lastmodifieddate value. Then, in the SF read snap you can add a filter to get the records greater than the lastmodified date. 

    This is the expression added in where clause of SF read, where $lastmodifieddate is coming from file/table  before SF read snap. In case you prefer using file, you can store the file in local sldb or any location that is available to you.

    Hope this helps !

    Thanks!