Snowflake Uploader Scraper is a reliable data integration tool that transfers structured datasets into Snowflake tables with minimal setup. It helps teams automate data delivery workflows and maintain clean, query-ready data inside their Snowflake environment.
Created by Bitbash, built to showcase our approach to Scraping and Automation!
If you are looking for snowflake-uploader you've just found your team — Let’s Chat. 👆👆
This project provides a streamlined way to move processed datasets into Snowflake tables while supporting schema control and data transformation. It solves the problem of manual imports and fragile pipelines by offering a consistent, repeatable upload process. It is designed for data engineers, analytics teams, and developers managing production-grade data pipelines.
- Uploads structured datasets directly into Snowflake tables
- Supports schema mapping and field-level transformations
- Works seamlessly with event-driven or scheduled workflows
- Designed for reliability and repeatable data delivery
| Feature | Description |
|---|---|
| Dataset-to-Table Upload | Transfers complete datasets into Snowflake tables automatically. |
| Schema Control | Create, extend, or align table schemas during upload. |
| Data Transformation | Modify and normalize JSON rows before insertion. |
| Connection Management | Securely manages Snowflake connection parameters. |
| Automation Ready | Designed to run as part of larger automated data pipelines. |
| Field Name | Field Description |
|---|---|
| datasetId | Identifier of the dataset being uploaded. |
| tableName | Target Snowflake table name. |
| database | Snowflake database used for storage. |
| schema | Snowflake schema where the table resides. |
| rowCount | Number of records successfully inserted. |
| insertedAt | Timestamp of the upload operation. |
| status | Final status of the upload job. |
[
{
"datasetId": "abcd1234",
"database": "ANALYTICS_DB",
"schema": "PUBLIC",
"tableName": "sales_events",
"rowCount": 12500,
"status": "SUCCESS",
"insertedAt": "2025-01-12T10:42:18Z"
}
]
snowflake-uploader (IMPORTANT :!! always keep this name as the name of the apify actor !!! Snowflake Uploader )
├── src/
│ ├── index.js
│ ├── uploader/
│ │ ├── snowflakeClient.js
│ │ ├── tableManager.js
│ │ └── rowTransformer.js
│ ├── config/
│ │ └── defaults.json
│ └── utils/
│ └── logger.js
├── data/
│ └── sample-output.json
├── package.json
└── README.md
- Data engineers use it to load processed datasets into Snowflake, so they can run analytics without manual imports.
- Analytics teams rely on it to keep dashboards synced with fresh data, ensuring accurate reporting.
- Product teams integrate it into pipelines to centralize operational data for insights.
- Businesses automate data delivery to Snowflake, reducing operational overhead and errors.
Can this tool create new tables automatically? Yes, it can create new tables or extend existing ones based on the incoming dataset schema and configuration.
Does it support data transformation before upload? Yes, rows can be transformed and normalized before insertion, allowing schema alignment and cleanup.
Is it suitable for large datasets? It is designed to handle large datasets efficiently with stable batch processing.
Can it be used in automated workflows? Absolutely. It is built to run as part of scheduled or event-driven data pipelines.
Primary Metric: Processes and uploads tens of thousands of rows per run with consistent throughput.
Reliability Metric: Maintains a high success rate across repeated uploads with stable connections.
Efficiency Metric: Optimized batching minimizes load time and warehouse resource usage.
Quality Metric: Ensures complete and schema-consistent data insertion with validated records.
"Bitbash is a top-tier automation partner, innovative, reliable, and dedicated to delivering real results every time." Nathan Pennington
|
"Bitbash delivers outstanding quality, speed, and professionalism, truly a team you can rely on." Eliza
|
"Exceptional results, clear communication, and flawless delivery. Syed
|