Parsing XML Data and formatting it

[acmohan023]

Hi Team,

We have a requirement to parse the xml file (1.5GB) and transform/group the content based on the one of the field value and write write multiple files based on the each group.

=== Sample Input ==

<?xml version="1.0" encoding="UTF-8" ?>

test

test

Test1 Test2 Test1

===== output File 1===

<?xml version="1.0" encoding="UTF-8" ?>

test

test

Test1 Test1

===== output File 2===

<?xml version="1.0" encoding="UTF-8" ?>

test

test

Test2

I have tried using the xml parser , split based on child and add headers back. Problem here is as it is huge data CPU and memory are going high and getting Connection lost error.

Have also tried xslt but still got same issue.

Can you please help me to design the solution with memory optimization.

Thanks in advance.

[acmohan023]

Team,

Can you please help and let me know if any details required.

[acmohan023]

@rsramkoski ,

can you please guide to resolve this issue.

[RogerSramkoski]

@acmohan023 - Thank you for providing the sample input and output. Since the post is about CPU and memory issues, could you please share a sanitized version of your pipelines as well. By sanitize, I mean removing values from any field that reveal SFTP hostnames, IP addresses, accounts, or other sensitive details - it’s just the overall logic I would like to see.

[acmohan023]

Hi @rsramkoski - Sorry for the delayed response. I have tried making the replica of pipeline used. Please find the attached pipeline for the scenario mentioned.

FileReadAndProcess_2021_08_23.slp (20.0 KB)
ProcessGroup_2021_08_23.slp (11.9 KB)
WriteGroupedData_2021_08_23.slp (5.1 KB)

[acmohan023]

Team,
Can you please help on this.