04-27-2023 09:44 AM
Is it possible to truncate a table in bigquery and then run Bulk Load(Streaming). I’ve tested using BigQuery Execute with truncate statement and then link a bulk load(streaming) snap. I get an error message saying invalid input. Its not recognizing the table schema?
04-27-2023 09:51 AM
Truncation followed by a load is very discouraged in BigQuery as the truncation is an operation that results in eventual consistency issues in the BigQuery backend (it needs to persist the state of truncation across all backend nodes). Our BigQuery Write and BigQuery Bulk Load (Streaming) will explicitly fail if the table is truncated as it may potentially cause data loss, you can try the BigQuery Bulk Load (Cloud Storage) as it has more appropriate retry mechanisms built in to support truncated table operations.
04-27-2023 10:31 AM
Ok makes sense. Are there other ways/alternatives to truncate and load a bigquery table in snaplogic?