Read a file from one project into a pipeline in another project

[vsunilbabu]

I have a file Account.csv in project space called landing. Want to use above file in a pipeline in another project space called staging.

Have found this article:
https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438844/File+Reader?showComments=true&show...

where I can use HDFS in expression of Mapper and map it as $FileName and then use File Reader.

Is this possible?

[psathyanarayan]

Files between two different project spaces cannot be shared in pipelines. However files within the same project space as that of the pipeline can be shared and also files that are in the project shared space can be shared and so are the files that are in global shared org. space can be shared.

In your given org. space let’s say the name of the org. space is org1. If a pipeline is created in /org1/projects/shared folder, then the pipeline can read files that are in /org1/projects/shared and in /org1/shared folder.

If the pipeline is created in /org1/shared folder, then it can only access files in /org1/shared folder.

If the pipeline is created in /org1/projects/staging, then it can access files in /org1/projects/staging, /org1/projects/shared and /org1/shared folder.

So to answer your question, the pipeline in project space “staging” cannot access the file “Account.csv” in project space “landing”

I think this was mainly done as a security feature so that different projects cannot read off the data from some other project.

However, one option I would suggest is changing the permission level for each of the project spaces allowing full access to the project space that is trying to access it.

Another way would be is to try and get your file into the global shared or the projects/shared area so that the staging project space can access it.

Still another approach would be is to try and use s3 and have the same credentials access the bucket from the pipeline.

Hope this helps.

[cjhoward18]

@psathyanarayan @vsunilbabu

Actually files can be read from a pipeline in a different project than the file being read, and does not need to be in the shared folder.

If you have the full absolute path/ accurate relative path to the file you are trying to read then there is nothing preventing you from doing so. However the File Reader snap will only recommend paths to files in the same project or files in the shared folder. Screen shot below shows an example of a pipeline reading a file from a different project, not in the shared folder, and how this is achieved. In this case the pipeline is in project ExpressionTester/shared and reading from a file in projects/alpha

[psathyanarayan]

Thank you @cjhoward18, but does it not require permission at the project folder level wherein we provide read, write, execute, owner or full access permissions. I thought by default we don’t give read permission across project folders more out of security.

[cjhoward18]

Yes, your right there are ways to restrict access rights for users over a certain project, so they would need to have access to both projects.