Forum Discussion

sathishmadavali's avatar
sathishmadavali
New Contributor II
3 years ago

How to add New Column data in to S3 Bucket

Hi, I have 3 Fields in S3 Bucket(eid,dname,dob) and SLDB Having 4 Fields( eid,dname,dob,ename). i have to load newly added Field data into S3 bucket. Thanks M SATHISH
mina's avatar
mina
Employee
5 years ago
Copy LinkReport

Hi Scott and @Supratim I’d think our Diff snap would also work. Here is the sample pipeline. What do you think?

  • tstack's avatar
    tstack
    Former Employee
    8 years ago
    Copy LinkReport

    What are the file formats (e.g. CSV, JSON, XML, …)?

    Do you need to do any transformations to the values before insertion?

    The DB Insert snaps (e.g. MySQL Insert) have an option to create the table if it is not present. I think the table creation is done based on the values in the first record to be, so string properties will be VARCHAR columns in the DB, numbers will be a number column, and so on. Is that sufficient or are you looking for more?

    • aleung's avatar
      aleung
      Contributor III
      8 years ago
      Copy LinkReport

      Auto create table with the perspective data typing and field length dynamically isn’t something you can easily do today. My experience with Redshift so far is Snap would default all fields to varchar and having max length of 32000 which isn’t ideal. What you can do is perform a dynamic max(length([field]) via SQL to obtain the length the field should be then try to port the data from the original table to the ideal one.