Pattern to ingest data from Google Storage into Google BigQuery using AVRO file format

Created by @pkoppishetty


This pipeline is designed for transferring data from a local machine to Google Storage, and then loading data into Google BigQuery using a data source as Google Storage.

37%20PM

Configuration

Make sure- Upload type is:- “Upload existing files from Google Cloud Storage” and file paths:- “file format”:“AVRO”, “file path”:“gs://gcs_cloud_1/data/leads.avro”

Sources: Local machine, Google Storage
Targets: Google BigQuery
Snaps used: File Reader, File Writer, Google BigQuery Bulk Load (Cloud Storage)

Downloads

Pattern to ingest data from Google Storage into Google BigQuery using AVRO file format.slp (5.6 KB)