Merging fields of a document

[aspanda]

I am extracting all fields of a object in salesforce. since it is huge we have to use two salesforce read snap to get all the fields . so i have used join to perform Inner join & combine all the fields, provided id as common field for both left and right path. But for huge amount of documents join snap kind of holding up all documents and then performs the inner join operation . due to this hold up it’s consumes huge memory. Could you please provide a better solution for this ?

[nsmith]

Don’t query all of the data at the same time. Have a pipeline execute that give parameters, so that your two Salesforce snaps are only selecting and joining a small subset of data (Use SystemModStamp?)

[Supratim]

@aspanda
Before joining use sort snap to improve performance.