SFTP errors
FYI I was getting errors with pipelines using a directory browser with a basic auth account for SFTP. After doing some testing and verifying the authentication details I ended up rolling back the Binary Snap Pack version to Main19844. Problem was resolved - must be a bug in new version. The error I was getting is Failure: Unable to create filesystem object for sftp://sftp.accorplus.com:2222/, Reason: Failed to get SFTP session connected, Resolution: Please check all properties and credentials
Move/renaming files on SFTP server
I’m designing a pipeline to get a data file from an sftp server, use the data, and then rename the file to a filename with date and time and move it to an archive subfolder. I am attempting to use the pipeline from the showcase at: Move/rename script on SFTP via script Designing Pipelines Here is a pipeline that renames/moves to a different location on the same SFTP location. This uses script snap. SFTPMoveFileScript_2017_11_09.slp (8.2 KB) Is there a better way to do it now? In the sample pipeline from the showcase it includes the username and password as variables rather than snaplogics authentications as used by the File Writer snap. Is there a way I can get this pipeline to use snapalogic authentication? If not, I see that the password is encrypted (obfuscated)… What is the best way to securely pass the password into the script? Thanks, Scott
Ingest data from File Server (FTP/sFTP) into AWS Cloud Storage (S3)
Contributed by @SriramGopal from Agilisium Consulting The pipeline is designed to transfer files from a FTP/SFTP server and load them to cloud storage (Amazon S3 in this case). This use case is applicable to Cloud Data Lake initiatives. This pipeline also includes, the Date based Data Partitioning at the Storage layer and Data Validation trail between source and target. Control Table - Tracking The Control table is designed in such a way that it holds the source load type (RDBMS, FTP, API etc.) and the corresponding object name. Each object load will have the load start/end times and the records/ documents processed for every load. The source record fetch count and target table load count is calculated for every run. Based on the status (S-success or F-failure) of the load, automated notifications can be triggered to the technical team. Control Table Attributes: UID – Primary key SOURCE_TYPE – Type of Source RDBMS, API, Social Media, FTP etc TABLE_NAME – Table name or object name. START_DATE – Load start time ENDDATE – Load end time SRC_REC_COUNT – Source record count RGT_REC_COUNT – Target record count STATUS – ‘S’ Success and ‘F’ Failed based on the source/ target load Partitioned Load For every load, the data gets partitioned automatically based on the transaction timestamp in the storage layer (S3) Configuration Sources: FTP/sFTP File Extracts Targets: AWS Storage Snaps used: File Reader, File Writer, Mapper, Router, JSON Formatter, Redshift Insert, Redshift Select, Redshift - Multi Execute, S3 File Writer Downloads IM_FTP_to_S3_load.slp (15.3 KB)Using "sftp -i [private key location] username"
A vendor is having us retrieve their info using this SFTP method: sftp -i /path/to/private/key username@vendor.net Any thought how to do this in SnapLogic? Do I need to do it as a script? Also I have a passcode that i will need to use? Thoughts? Thanks, Bob
SFTP file reader for daily pulls
Hello All, We are going to start receiving delta files to a data warehouse and this is the format that our vendor specified that they will be sent over. Company_Data_Storage_TITLE_YYYYMMDD_HHMM_YYYYMMDD_HHMM.txt My question, in the file reader how would I format it so that it’ll be able to pull every day without having any issues.
