Failed to write to file
I am able to read input files from a drive, where as while writing it to the same drive i have the below error. I have both read & write access to the drive.
Consolidated file - File Writer[630c7b8...
Forum Discussion
I should probably have asked if you know what type of data is stored in that BLOB structure. When you try to compress data that is already compressed (e.g. image/audio/video), the compression won’t gain much and sometimes gets a bit bigger. So in this case, it appears that the data is likely already compressed.
You may need to find another storage model for this data outside Snowflake if it doesn’t fit in the max size field. As @ptaylor recommended, you should probably consult the Snowflake community for suggestions there.
Thanks Kory. We are exploring various other options with Snowflake cust support.
Hi @shawnharv,
Try and wrap the payload into one object, and pass that object into the Entity field in the REST Post snap.
Something like this:
[
{
“payload”: {
“kind”: “doubleclicksearch#conversionList”,
“conversion”: [
{
“clickId”: “XXXXXXXXXXXXXXXXXXMYaAunkEALw_wcB”,
“conversionId”: “XXXXXXXXXXXzhXE5AAM”,
“conversionTimestamp”: “1623361621000”,
“segmentationType”: “FLOODLIGHT”,
“segmentationName”: “SFDC_Wins”,
“type”: “ACTION”
},
{
“clickId”: “XXXXXXXXXXXXd69532af3081b22f4”,
“conversionId”: “XXXXXXXXXXXXXzhW2pAAE”,
“conversionTimestamp”: “1623352858000”,
“segmentationType”: “FLOODLIGHT”,
“segmentationName”: “SFDC_Wins”,
“type”: “ACTION”
}
]
}
}
]Don’t forget the Content-Type Header:
Thanks [bojanvelevski] for the quick response.
I tried wrapping this in the payload object like you suggested, but still no success. The doubleclick search API responds with “The request was not valid. Details: [Empty batch encountered]”.
Its very strange how the payload works when its coming from a JSON generator snap, but the same JSON payload coming from a file fails. The two are literally the same haha.
Thanks,
ShawnPlease review and post a screenshot of the Pipeline Execution Statistics of each version of the pipeline – the one with the JSON Generator and the one without. This will give us more information about exactly how many input documents are being received by the REST Post.
