Forum Discussion

JmPearl's avatar
JmPearl
New Contributor II
3 years ago
Solved

RPE only returns small proportion of documents

I have a RPE running with the snaplogic recommended data science docker image. When i pass documents to the remote python script (hundreds, sometimes thousands) I only get around 1% of the documents back from the snap. Even when i literally pass the documents through the snap without any transforms, I get only 1% back.

whats going on here?
Any help with this is appreciated

python
  • JmPearl's avatar
    JmPearl
    3 years ago

    Ultimately i figured it out by digging into the log files produced by my RPE. The text being sent to the remote python code snap was not decoded properly, and therefor the RPE would break at the nth document which had faulty text. The problem was fixed by setting up a mapper snap that used HTML.decode on the text prior to the RPE snap.

4 Replies

  • JmPearl's avatar
    JmPearl
    New Contributor II
    3 years ago

    More detail: this seems to only happen when i include a text field in the documents passed to the RPE

  • apranay's avatar
    apranay
    Employee
    3 years ago

    I tried RPE snap but I got all documents in snap output.

    Can I know is there any specific data or fields.

  • JmPearl's avatar
    JmPearl
    New Contributor II
    3 years ago

    Ultimately i figured it out by digging into the log files produced by my RPE. The text being sent to the remote python code snap was not decoded properly, and therefor the RPE would break at the nth document which had faulty text. The problem was fixed by setting up a mapper snap that used HTML.decode on the text prior to the RPE snap.

  • koryknick's avatar
    koryknick
    Employee
    3 years ago

    @JmPearl - this may have been a faster diagnosis if you enable both the error view and second output view of the Remote Python Script snap. But checking your script logs was a great choice.

    Expected output:

    • First output view: The output from the Python script’s execution.
    • Second output view: The STDOUT redirected from the RPE.