Forum Discussion

Contributor

7 years ago

Limit/Restrict the CSV parser output instead of scanning entire file

Is there is any possible ways to limit the CSV parsing output?

Say for example, I have 30 + GB of CSV file, from this file I need to get only first two lines of data
i.e. The first line is header and second line is data. I hope skip lines wont’t work for this approach.

Can any one suggest instead of parsing/scanning the entire 30 + GB of file how to get first two lines from the file with best optimized approach?

Thanks in advance!

1 Reply

Parin
New Contributor
5 years ago
Hi Alchemiz,

Thank you for the sample snap and your assumption is correct but this is not the exact structure that i am looking for.

The different records are part of same array like in case of fname.

I have attached a smaple json for mutiple records which i need to create.

Also is there a way we can use JSON Generator snap which can use csv file data and convert it into JSON as i looking for something more dynamic.SampleJSON.txt (1.9 KB)
- Supratim
  Contributor III
  5 years ago
  
  Parin:
  
  [
  {
  “attributes”: {
  “Name”: [
  {
  “value”: “Test1234”
  }
  ],
  “Addresses”: [
  {
  “value”: {
  “AddressLine1”: [
  {
  “value”: “122 Test St”
  }
  ]
  }
  }]
  }]
  
  @Parin
  Use mapper snap and keep your desired structure. Only replace the hardcoded part with $ like below.Then use group by N and make group size 0 to make it as array
  
  {
  “attributes”: {
  “Name”: [
  {
  “value”:$Name
  }
  ],
  “Addresses”: [
  {
  “value”: {
  “AddressLine1”: [
  {
  “value”: $AddressLine1
  }
  ],
  “City”: [
  {
  “value”: “AVC”
  }
  ],
  “StateProvince”: [
  {
  “value”: “WV”
  }
  ],
  “Country”: [
  {
  “value”: “US”
  }
  ],
  “Zip5”: [
  {
  “value”: “25271”
  }
  ],
  “AddrBestFlag”: [
  {
  “value”: “Y”
  }
  ],
  “Zip4”: [
  {
  “value”: null
  }
  ]
  }
  }
  ]
  }
  }