Great question! The perfect solution for your use case is the Union Snap. Here's a breakdown of how to approach this:
---
✅ Using the Union Snap as a "Binary Union"
The Union Snap merges two or more document streams into a single output stream — exactly what you need to avoid duplicating your File Writer Snap.
---
🔧 Recommended Pipeline Structure
File Reader
└── Router
├── [Direct / same format] ──────────────────────────────┐
│ ▼
└── [Excel/CSV → Parquet] → Parquet Formatter ──→ Union ──→ File Writer
Step-by-step:
1. File Reader reads the source file.
2. Router splits the stream based on your source/target configuration:
- Route 1 – Direct pass-through (source = target format, no conversion needed)
- Route 2 – Conversion path (e.g., Excel/CSV → Parquet formatter)
3. Union Snap merges both routes back into a single stream.
4. File Writer receives the unified stream and writes the output.
---
⚠️ Key Considerations
| Consideration | Detail |
|---|---|
| Stream consistency | Both inputs to the Union should ideally have a consistent document schema. If the direct path and the converted path produce different structures, downstream issues may arise. |
| Order not guaranteed | Union simply appends streams — it does not guarantee document ordering across the two inputs. |
| No join condition needed | Unlike the Join Snap, Union requires no matching key — it just concatenates the streams, which is exactly what you want here. |
| Multiple input views | The Union Snap supports 2 or more input views, so you can extend this pattern if you add more conversion routes later. |
---
💡 Tip
Since your Router is coming off a File Reader (binary → parsed documents), make sure your conversion path (e.g., CSV/Excel → Parquet) completes its formatting before feeding into the Union, so both streams are in a compatible document format when they arrive at the File Writer.
This is a clean, scalable pattern — one File Writer, multiple upstream paths! 🎯
