Forum Discussion

LordNeilLord's avatar
LordNeilLord
New Contributor II
3 years ago
Solved

Mixed array and object JSON

Hello, I have a feed coming from github which I need to get certain attributes but some of the items are being returned as arrays and some are being returned as objects! Does anyone know how I can ...
  • snapation6713's avatar
    snapation6713
    2 years ago

    koryknick,   Thank you for your help.  I was provided a solution by a coworker.

    It's essentially that pipeline solution you provided HTTP Client Pagination  with a few tweaks.  I modified the HTTP Client as follows: Removed pagination configurations and the parameters configurations for both http snaps and added to the body of the 2nd HTTP snap   "after" : $splitValue, "limit" : 100    Then I was getting subsequent data.   But the data was duplicating.  So I modified the map for pagination as follows:  

    sl.range(1, parseInt($entity.total), 100)  -I also changed the 1st http snap to just give me 1 record just so that I get the total  and I process from page 1 to the end in the rest of the pipeline so I don't do a union.  
     
    And it worked!   I am now getting subsequent data and it the records are not duplicating.  

     

     

jovan_j's avatar
jovan_j
New Contributor III
6 years ago
Copy LinkReport

Hi Harsha,

Have you tried using the BINARY response type in the REST Get snap:

The output that this produces is:

Let me know if you need anything different.

Regards, Jovan.

Harsha3's avatar
Harsha3
New Contributor III
6 years ago
Copy LinkReport

Hi Jovan,

Thanks for writing may I know whether is there any up gradation done because this snap was working fine 5 months before .

  • SpiroTaleski's avatar
    SpiroTaleski
    Valued Contributor
    5 years ago
    Copy LinkReport

    Hi @amubeen,

    After CSV parser snap, put the mapper snap with the fields that you want to extract from the file. When you connect CSV parser with the mapper and validate the pipeline, you will get all the fields from the file in the “Input Schema” of the mapper(marked with blue). From there you can drag the required fields and drop to the expression field.

    Ex:

    In the Target Path, you should specify field names from snowflake database table(marked with black).

    You can get all the fields in the mapper “Target Schema”(marked with yellow) from the selected Snowflake table, if you connect the mapper with the downstream Snowflake Insert snap, and validate the pipeline.

    Hope this helps.

    Regards,
    Spiro