Forum Discussion

acesario's avatar
acesario
Contributor II
5 years ago
Solved

Using variable for file naming only and not in xml payload?

I’m formatting and writing an xml file, and would like to name the file based on a field value from the original file, but cannot be part of the xml. Is it possible to somehow bring a variable ($VAR) from a prior step in to the File Writer without including it in the preceeding XML?

‘response-s-’$VAR’+‘-.xml’

pipeline
  • darshthakkar's avatar
    darshthakkar
    9 months ago

    arvindnsn - I tried with a sample pipeline that generates 6 files over the SLDB and the performance is pretty good. It took ~1.5 mins for completion.
    I didn't connect to SFTP but just to add complexity on the same lines, I added an email snap that sent me a copy of those files, screenshots below:

     

    I have used the innate FileDelete snap for deleting multiple files at once (which does process files one by one)

    I do agree that if you're having a huge volume, this process would have to run over the weekend. Please lemme know your volume and frequency of this run to understand the issue in detail and whether we can scale up the solution.

    Thanks.

    Best Regards,

    Darsh

5 Replies

  • Supratim's avatar
    Supratim
    Contributor III
    5 years ago

    @acesario You can segregate xml format and file writer in different pipeline, then call this child pipeline using Pipeline execute and pass input param(the filename you want ) from your parent pipeline. Even if you use xsd inside xml format, you can found the xsd structure from the mapper(Map to xml) of your parent pipeline.

    • acesario's avatar
      acesario
      Contributor II
      5 years ago

      @Supratim Thanks for the quick response! I actually had tried using a separate pipeline, and was able to pass the parm as you suggest, however my xml output would be one row, rather than the aggregated rows. So, a. I was doing this before the map to xml. I could try it after that.
      Or better yet, b. Is there a way to have sub pipeline produce the aggregated rows, just like the parent does?

      What I mean is, Parent produces xml:
      <#root>
      <#row1>
      <#row2>
      <#row3>
      <#/root
      …
      Child (sub) currently produces
      file1…
      <#root>
      <#row>
      <#/root>
      file 2…
      <#root>
      <#row>
      <#/root>
      file 3…
      <#root>
      <#row>
      <#/root>

      • Supratim's avatar
        Supratim
        Contributor III
        5 years ago

        Actually you pass each row from pipeline execution snap, Move mapper to child pipeline , it should work.
        Unless the upstream snap of mapper produce multiple output of row.

  • acesario's avatar
    acesario
    Contributor II
    5 years ago

    Yes, the upstream snap produces multiple rows.

  • acesario's avatar
    acesario
    Contributor II
    5 years ago

    @Supratim, I found a way to resolve this. I used a GATE snap on the parent to combine the rows, and a json splitter on the child along with the parameter for the file naming.
    WORKS PERFECTLY! Thank you for the continued conversation and ideas.