Forum Discussion

Trakmaker's avatar
Trakmaker
New Contributor
4 years ago

Rest PUT - Error when using upload files field

Hi everyone, I am able to upload incoming data into blob storage with the below pipeline but not the format I am expecting. I need to do it this way: Get Data ->CSV Formatter->File Writer-> ...
acmohan023's avatar
acmohan023
New Contributor
5 years ago
Copy LinkReport

@rsramkoski - Have attached the sample input structure and output structure for your reference. (I had pasted request and response structure and it is not visible).
InputAndOutpultSample.txt (1.3 KB)

As file Size is huge , tried to split the flow into multiple pipelines (to process it in chunk and release memory) :

  1. Read the file from SFTP and XML Parse to split the data . Group by N and call 2nd pipeline
  2. split group , by sort by field ( input field in attached file) call 3rd pipeline
  3. write files smaller chunk to local sftp
  4. On completion of all above 3 pipeline , read small chunk files based on file name and write consolidated file per group by field (input field in attached file)

Please suggest if there is any other approach to resolve this issue.

RogerSramkoski's avatar
RogerSramkoski
Employee
5 years ago
Copy LinkReport

@acmohan023 - Thank you for providing the sample input and output. Since the post is about CPU and memory issues, could you please share a sanitized version of your pipelines as well. By sanitize, I mean removing values from any field that reveal SFTP hostnames, IP addresses, accounts, or other sensitive details - it’s just the overall logic I would like to see.

  • SpiroTaleski's avatar
    SpiroTaleski
    Valued Contributor
    4 years ago
    Copy LinkReport

    @SGArchitect

    That is happening because the output of the Union Snap will produce multiple documents(from both inputs). And some of the documents does not contains the value that you are referring to, which throws an error.

    By checking the “null safe acess” in the mapper snap will prevent the error, and on the output of the mapper snap you will have the same number of documents as input. The documents that does not contains the path that you are referring will have “null” value on output.

    BR,
    Spiro Taleski

  • alex_panganiban's avatar
    alex_panganiban
    Contributor
    4 years ago
    Copy LinkReport

    I think as an alternative to using the null safe access feature, you could also use the hasPath() and get() methods. I like using the get() a little more because if the path doesn’t exist, then it will automatically assign a null to the result of the expression, whereas the hasPath() method renders a true or false result. If you use hasPath(), then you’ll probably want to incorporate it as part of a ternary expression.

    Here’s a few conceptual examples for you.