Forum Discussion
I assume you are using the JSON Formatter to write the file, yes? But how are you formatting it? And what application are you using to try to open the file?
By default, the JSON Formatter will use a very compressed format with no line breaks. Some editors don’t deal well with a file where all the data is on one very long line.
You could enable “Pretty Print” on the formatter, which produces a much more readable and verbose format like this:
But this format can also be a challenge for some JSON-capable applications since all of the data is inside a single JSON array.
You might want to consider the “JSON Lines” format, where each line is a compactly formatted JSON object representing one document:
This is often the best choice when dealing with a “log”. Each JSON line corresponds to a line of your CSV file. See https://jsonlines.org/
If you are using Google/Gmail as your mail provider, Gmail’s Push Notification feature can utilize the Cloud Pub/Sub API to instruct Gmail to publish email deliveries to a Topic that match a certain condition (e.g. the new email has a particular label/in a subfolder). Outside of that, you would have to go back to polling the Users.messages:list API and managing the state yourself.
If using the Pub/Sub API, subscribers to a topic can then either pull and acknowledge messages, or in turn push the message(s) to another endpoint.
Pulling from that topic subscription can be done with scheduled tasks, and the REST Snap Pack with OAuth 2.0 Account.
First, the prerequisites defined by Google must be fulfilled:
- Create a new project in the Google Cloud Platform Console
- Enable Billing
- Enable the Cloud Pub/Sub API and the Gmail API
- Create new OAuth 2.0 Web Application API credentials under Identity and Access Management (IAM) → API Credentials
-
Create a Topic and grant
Pub/Sub Publisher
permission toserviceAccount:gmail-api-push@system.gserviceaccount.com
-
Create a Pull Subscription on that Topic and grant
Pub/Sub Subscriber
to the email account(s) you want subscribed to the email notifications.
Setting up the OAuth 2.0 Account is very similar to the instructions in the Connecting SaaS Providers with SnapLogic’s OAuth-enabled Snaps blog post. The relevant
scope
values are “https://www.googleapis.com/auth/gmail.readonly https://www.googleapis.com/auth/pubsub
”.Then a
watch
request needs to be executed semi-regularly on the topic to maintain the subscription:
robin-community-gmail-watch_2017_04_14.slp (4.2 KB)Another pipeline can then poll the subscription, and receive notifications when changes have occurred with the topic. The issue here is that Gmail’s API is designed for synchronization - meaning the notifications the topic receives are
historyId
s (just pointers to the fact that something changed).To find out what changed, a cache of previous
historyId
s needs to be maintained and used to query thehistory.list
API. For each history event (e.g.messagesAdded
), you then need to query theUsers.messages:get
API to get the actual email content. Then the cache should then be updated with the newly receivedhistoryId
.Since effective use of GMail requires some of level of tracking state, this can result in a pipeline that is a little busy but the following crude example shows that it is possible to use a pipeline to read new email messages:
robin-community-gmail-pull_2017_04_14.slp (26.7 KB)As for utilizing a Push Subscription model for your preferred non-polling solution, this would obviously suit an Ultra pipeline very well but is complicated by Google’s restrictions on only publishing to a domain that is owned and controlled by you and to an endpoint secured by a non-self-signed SSL/TLS certificate.
I haven’t investigated this yet but I imagine this would be possible by setting up a custom API Gateway (e.g. using App Engine) or reverse proxy registered and available at a domain you control, that handles the secure redirection to the Ultra pipeline URL.