omniload is a polyglot data loader framework based on dlt. It allows you to load data from any source into any destination, either using a concise CLI from your shell, or the Python API from your own applications.
omniload provides the same efficient incremental data loading modes
inherited by dlt: append, merge, and delete+insert.
We recommend using uv to install or run omniload.
pip install uv
uvx omniloadAlternatively, if you'd like to install it globally:
uv pip install --system omniloadWhile installation with vanilla pip is possible, it's an order of magnitude slower.
The next command instructs omniload to read the table public.some_data from
your PostgreSQL instance, and to write the data to your BigQuery warehouse
under the schema omniload and table some_data.
omniload ingest \
--source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
--source-table 'public.some_data' \
--dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
--dest-table 'omniload.some_data'Please visit the full documentation, or inspect the list of supported sources and destinations.
Contributions are very much welcome. Please visit the Documentation to learn how to spin up a sandbox environment on your workstation and submit patches, or create a ticket to report a bug or propose a feature.
Breaking changes should be expected until a 1.0 release, so version pinning is strongly recommended, especially when using this software as a library. For example:
pip install 'omniload[full]==0.0.42'The project is licensed under the MIT License, see the LICENSE file for details. Some components are licensed under the Apache 2.0 license, see the NOTICE file for details.
This project would not have been possible without the amazing work by the authors and contributors to SQLAlchemy, dlt, and ingestr, turtles all the way down. Kudos.