File Reader doesn't always return the entire file

[plancor]

Hello, I have a simple pipeline where the first snap is a File Reader.  It should read a ~3.3M file using SFTP.  Sometimes it reads the entire file, and other times it partially reads the file.  I ran the pipeline twice three minutes apart.  The first execution read 1.6M and the second execution read the entire file.  Why would this occur, and is there a way to ensure the entire file is read?  Thank you

[koryknick]

@plancor  - is it possible that the file is still being written when the SnapLogic pipeline starts?  Or is the pipeline failing and aborting the read?

There is nothing in the File Reader itself that would stop reading until it hits the end of file marker.

[plancor]

Thank you, Kory.  Do you have a suggestion on how to handle this issue?

[koryknick]

@plancor - the simplest solution I've used is to have the job that is writing the file create it using a temporary file name that doesn't match the pattern that SnapLogic is looking for, and rename it when the file is complete.  Or if you are using any kind of enterprise scheduling tool, you can make the SnapLogic job dependent on the job creating the file.

[plancor]

Hi Kory, the file is always complete before the pipeline starts.  The problem is that the File Reader snap does not always read the entire file.  In fact, it does not read the whole file most times...yet sometimes it does.