Forum Discussion

msalter's avatar
msalter
New Contributor II
4 years ago
Solved

Lost contact with Snaplex node

I have a very small pipeline (3 snaps) that I’m reading from a SQL table and writing the results to a new table in the same DB. I keep getting an error: Lost contact with Snaplex node while the pipe...
  • koryknick's avatar
    3 years ago

    @darshthakkar - I believe @dmiller is correct that the Shard Offsets is a custom snap. I’ve replicated the logic with core snaps and built in expressions. See the attached example.
    Shard Example_2023_07_05.slp (5.6 KB)

    The Mapper is where the magic happens.

    sl.range(0, $TOTAL, Math.ceil ($TOTAL / parseInt(_shard_count)))
    .map((val,idx,arr)=> 
      idx == arr.length - 1 
      ? { offset : val, limit : $TOTAL - val } 
      : { offset : val, limit : arr[idx + 1] - val } 
    )
    

    First, use the sl.range() built-in function to generate an array with the offsets to be used, then use Array.map() method to recreate the simple array of integers as an array of objects to provide both the offset and limit for each shard.

    After the Mapper, just use a JSON Splitter to get a new document for each limit and offset combination (same as Shard Offsets snap) that would feed your child pipeline with the Pipeline Execute.

    Hope this helps!