Forum Discussion

leonardoamorim's avatar
leonardoamorim
New Contributor
5 years ago

REST Get Pagination

Hi everybody,

I’ve been thinking about a solution to my problem the past few weeks and still couldn’t find a proper solution.

The problem is: I have an REST Output which gives me the following.

The total of outputs I have for ONE day and the data with a limit of 100.
It doesn’t have the offset or number of pages in the output.

I want to fetch all the data for a day (in this case, 10811) so I created a MAPPER with the following values before asking for the REST GET with PAGINATION

Then, I do a REST GET with the following values for:

  • from
  • to
  • type
  • limit (always a 100)

The problem is when I try to do the pagination that does not work.

For the HasNext field: $entity.data.length > 1 (this is always true until it reaches the last offset value and it returns me an array with 0 data)
For the NextUrl fiel i have : “URL” + + “?from=” + $from + “&to=”+ $to + “&type=”+ $type + “&limit=”+ $limit + “&offset=” + (snap.out.totalCount+100).toString()

I need to increment the OFFSET in the REST GET by 100 every iteration, with a LIMIT of 100 outputs every time, until I have all the TOTAL outputs (in this case, 10811).

I don’t know what I am doing wrong, but i’m getting an error:
“error”:“not found: unable to find valid operation for: null”,“code”:“S21”}"

Can someone help me? Thanks =)

pipeline

4 Replies

  • dmiller's avatar
    dmiller
    Former Employee
    6 years ago
    Copy LinkReport

    I’m trying to understand the ask. Are you saying you want a kind of universal Mapper configured to accept n number of fields, but if a field is not in the input, not to write that field to the output (instead of writing it with a null value)?

    • philliperamos's avatar
      philliperamos
      Contributor
      6 years ago
      Copy LinkReport

      That’s correct.
      I essentially have 25+ data sets to extract from a source system, and from those data sets, there are about 50 common fields.
      Hence, a master mapper with the common fields, that will ignore the ‘missing’ fields.

      • cjhoward18's avatar
        cjhoward18
        Employee
        6 years ago
        Copy LinkReport

        In your example pipeline given, if you enable the Null-safe access in the mapper, then attach another mapper at the end with the expression: $.filter((value, key) => value != null) mapped to the target: $ will give you all the values of your common mappings that are not null.