Forum Discussion

darshthakkar's avatar
darshthakkar
Valued Contributor
3 years ago
Solved

Reading data from tsv file

Hi Team, How can we read data from tsv file? I have tried using CSV and Fixed Width Parser but no luck thus far. Thanking in advance for your help. Regards, Darsh
ddellsperger's avatar
ddellsperger
Admin
2 years ago

I'm not sure what exceeds_total_count is in this case, but you might be able to leverage that, if not you likely will have to check the size of data $entity.data.length for whatever page size you're requesting.

If, for instance your page size is 100, you could put $entity.data.length >= 100 in your has next, and then your next url would have to be something like $url + "&offset=" + 100 * (snap.out.totalCount + 1) since the next page is calculated before the output page is processed, so it will be one off. If on the other hand, you know how many pages you need specifically, you can set the has next to simply true and then set the page limit in the snap settings.

terrible_towel's avatar
terrible_towel
New Contributor II
2 years ago

I do know the total count as it comes back from each request. I've tried setting the has next to 'true' and then the total pages to fetch = &entity.total_count, but it complains that 'entity' is undefined for some reason. What am I missing?

  • eguo's avatar
    eguo
    Former Employee
    9 years ago

    Looks like you are facing a transformation issue. Normal snaps can do the job.

    Assume your input data is flat like what’s in following picture.

    They would looks like this in a snap preview:

    [
  {
    "S": "1",
    "STATUSDESCRIPTION": "Testing 1.",
    "P": "urgent"
  },
  {
    "S": "2",
    "STATUSDESCRIPTION": "Testing 2.",
    "P": "normal"
  }
]

    A pipeline like this will produce the expected output:

    The Group By N snap can group individual input records into an array.

    The Mapper snap should looks like this:

    • matt_bostrom's avatar
      matt_bostrom
      New Contributor II
      9 years ago

      this is perfect! on my health check call it was the same solution you provided. hopefully this helps others. @del I’m sure your solution works but this other one requires no code so good to know going forward!

      • del's avatar
        del
        Contributor III
        9 years ago

        Yep, that’s good to know. I guess I missed the release of the Group By N snap. I wouldn’t have had to develop my own 🙂 .

  • del's avatar
    del
    Contributor III
    9 years ago

    @matt.bostrom, The simplest way that I know of is to use a Script snap. I’m not a Python expert (so someone can improve on this), but code in the execute method would look something like this:

    def execute(self):
    tickets = []
    out = java.util.HashMap()
    while self.input.hasNext():
        try:
            in_doc = self.input.next()
            tickets.append(in_doc)
        except Exception as e:
            errWrapper = {
                'errMsg' : str(e.args)
            }
            self.log.error("Error in python script")
            self.error.write(errWrapper)
    out["tickets"] = tickets
    self.output.write(out)

    I’ve run into similar needs before on a couple of occasions, so I have written a custom snap to achieve this same result. I can share the java code with you as well if you want to develop your own snap.