All green but no upload file

I am doing a very simple pipeline that read one single file, then PGP encrypt it then SFTP to a remote server. When I execute the pipeline, I got all green but the file never got transmitted to the server.

Why? any hint?

If you are doing this in data preview mode you will want to check to make sure that the file writer is set to validate and execute

@mtran21, here are 2 hints, based on my experience:

  1. “all green” doesn’t necessarily mean your pipeline read or transmitted a file. Different snap configurations (error handling config, etc.) have different behaviors. You’ll need to check your image Pipeline statistics to see if a file was actually read and processed downstream in the pipeline.

  2. Some SFTP servers immediately move a file for processing after receiving it in the inbox. So, depending on the target SFTP server configuration, just because you don’t see the file in the inbox after you attempted to transmit it, doesn’t mean it wasn’t delivered. If the delivery was to an outside party system, you need to work with that party to see if they received it.

@mtran21 Can add output view of file writer and run the pipeline. If it’s working fine then can see one document count should comes from output view of file writer.

1 Like

On my Pipeline statistics, it shows all green. It even shows one file has been process.

But when I check SFTP server, the file is not there.

This SFTP server does not remove file. I can manually upload the file to the SFTP server just fine and the file I manually uploaded is still there.

Can you tell me more about adding output view? What snap do I need to add after File Writer?

Thanks

runtimelogs.txt (17.4 KB)

This is my runtime log.

@mtran21 Open file writer, then go to view tab click on + icon besides output as shown below. I believe you have used sftp url correctly like -
sftp://:// with valid account

So I added output view, I ran the pipeline again and it greened up. But the file never got transferred. I look at this, why is it the output0 file size is 0?

@mtran21 It’s strange. Just delete file writer snap and try with another new file writer snap. If possible share your pipeline.

@mtran21, you should not expect the bytes count in the output view - just the input view. The document count of “1” is expected; which is the result JSON document.

For further troubleshooting, I recommend changing the Snap Execution configuration on each of your snaps, including file writer to Validate & Execute

image

Then, validate the pipeline and inspect the output data from the validation.
When you inspect the output data, it should be similar to this:
image

In the above case, “overwritten” indicates the file was either written or overwritten. Of course, your filename will be different.

Maybe not, but I still suspect your target SFTP server is moving the file after it is delivered. Even if you test with an FTP client and can see the file in the inbox; give it a few seconds and then refresh the client and see if it’s still there. I have experience of this behavior with a certain bank/credit card company’s FTP server.

1 Like

We have faced some issues like this. There may be chances that your files are swept after snaplogic writes data into SFTP location. Please try to check SFTP/FTP owners if they found anything in their log. Try to copy the files in SLDB itself and see if files are written and have data in it.

Hope it helps!!

1 Like

Not sure what happened over night. My guess is that the FTP server admin did something and this morning I ran the pipeline again, the file finally got transferred. Not sure why but it’s working now. Thanks for all your help.

1 Like