Pipeline To Output one File per Date
I need to get Distinct update_dt and write one file per date from a oracle table sufixing the filename with date value. can you provide a sample pipeline which has similar functionality
I have crea...
Forum Discussion
@swright, if you refer to below pipeline. It will make two REST API call to two different end point (URL constructed using input) and with HTTP body content as you see in the example. So basically the REST snap will take care of processing each document for you.
Thanks Sripathi! That worked perfectly!
Scott
All my existing pipelines (which were functional in non-Ultra) were using XML Parser snaps already… My client program which was making requests to SnapLogic was setting the “Content-Type” header to “application/xml” which I suspect might have been causing this error.
I switched my client program to send a JSON payload and also modified the Content-Type header to send application/json and my Ultra pipeline accepts the payload.
Can you confirm if an Ultra pipeline that is sent an XML payload should also set the Content-Type header to application/xml or will that cause problems?
The Content-Type should always match what is in the payload.
@tstack, thanks so much for your input… so I’ve built the following pipeline which successfully executes in Ultra:
In my original pipeline I was passing in a URL parameter which was being captured as a pipeline parameter. In Ultra, I want to capture this pipeline parameter by passing a parameter to the “Pipeline Execute” snap in my pipeline. However, based on the documentation for Ultra… I have two questions:
- Are URL query parameters for an HTTP POST request with an XML payload accessible from within an Ultra pipeline? If so, how are they accessed?
- If the answer is no, then would it be possible to access the binary document’s headers from within the Ultra pipeline? If so how would this be done?
I think you can, but it’s not as easy as it oughta be right now. These values are available in the header of the binary document that is sent into the pipeline. Unfortunately, the XML Parser doesn’t give you access to the header. Instead, you can use a ‘Binary To Document’ snap to convert the binary document and its header into a regular document. By default, the body of the request is converted into a base64 string and placed in the ‘content’ field. The header data is added directly to the document. As an example, an ultra pipeline with a ‘Binary To Document’ snap and a Mapper that maps ‘$’ to ‘$content’ returns the following when a request is sent:
curl -k --data-binary '<foo><bar>hi</bar></foo>' -H 'Content-Type: text/xml' 'https://feed1:8082/api/1/rest/feed-master/queue/snaplogic/projects/shared/ts-xml-ultra-task?param1=value1' | jq .{ "task_name": "snaplogic/projects/shared/ts-xml-ultra-task", "content-length": "24", "method": "POST", "query": { "param1": [ "value1" ] }, "message_id": "8e22c67515a888254fdb859c090c2c6124bfafca-22392@feed1", "uri": "https://feed1:8082/api/1/rest/feed-master/queue/snaplogic/projects/shared/ts-xml-ultra-task?param1=value1", "content": "PGZvbz48YmFyPmhpPC9iYXI+PC9mb28+", "accept": "*/*", "client_port": 59455, "path_info": "", "host": "feed1:8082", "server_ip": "10.1.10.222", "content-type": "text/xml", "client_ip": "10.1.10.222", "server_port": 8082, "user-agent": "curl/7.45.0" }So, you can send the output of the ‘Binary To Document’ into the PipeExec, passing whatever values you want as pipeline parameters. In the child pipeline, it should start with a Document To Binary snap to get back the original XML stream of data and then an XML Parser to parse that.
Sorry, it’s a bit roundabout. The focus has mostly been on requests being JSON-based.
