GitHub - hendurhance/koel: 💱 Self served currency exchange rate api

💱 Koel is a free, open-source, self-hosted exchange rate API that aggregates real-time currency conversion data by scraping multiple sources, ensuring high availability and accuracy through an event-driven, fault-tolerant architecture.
No usage limits, no API keys. Works great on client-side in the browser or mobile apps. You can deploy it on your own server or in a Docker container.

Report Bug · Request Feature

If you are here to understand how this was built, you can check out the technical documentation for the system design, architecture, and design patterns used in this project. You can also check out the list of features to see what this project can do.

About Koel

Koel is a free, open-source, self-hosted exchange rate API that provides api endpoint to retrieve latest rates, historical data or time series data. It aggregates real-time currency conversion data by scraping multiple sources, ensuring high availability and accuracy through an event-driven, fault-tolerant architecture. Koel is designed to be easy to deploy and use, with no usage limits or API keys required.

Features

Real-Time Data Aggregation: Scrapes multiple sources to gather the latest exchange rates.
Self-Hosted: Deploy Koel on your own infrastructure without relying on third-party APIs.
Fault-Tolerant: Uses a factory pattern and multiple data sources to ensure data availability even if one source fails.
Event-Driven Architecture: Background tasks powered by Celery for regular and scheduled scrapes.
High Performance: Optimized for high-volume writes and fast read operations with techniques such as denormalization, bulk upserts, and partitioning.
No Usage Limits: Enjoy unlimited requests with no API keys required.
Client-Side Friendly: Easily integrate with browser or mobile applications.
Docker Support: Containerize your deployment for consistency and ease of use.

Getting Started

Prerequisites

Python 3.8+ – The project is built with Python and uses modern asynchronous libraries.
PostgreSQL 10+ – Recommended for its native partitioning support.
Redis – Used for caching, which is required for the application to run and keep track of jobs.
Celery – For background tasks and scheduling.
Alembic – For database migrations.
Docker & Docker Compose (optional) – For containerized deployment.

Clone the Repository

git clone https://github.com/hendurhance/koel.git
cd koel

Installation

Using Docker Compose

Build and Start the Containers:

docker-compose up -d --build
# or use the Makefile
make build && make up

The Docker setup includes:
- PostgreSQL: Database service with health checks
- Redis: Cache and message broker with persistence
- API: FastAPI application service
- Celery Worker: Background task processing
- Celery Beat: Scheduled task management
- Init: Automatic database migration and seeding
- Test: Dedicated testing environment
Access the API documentation:

http://localhost:8000/docs

Using Makefile Commands

The project includes a Makefile with convenient commands:

# View all available commands
make help

# Development workflow
make build          # Build Docker containers
make up             # Start all services with initialization
make down           # Stop all services
make logs           # View service logs
make shell          # Open shell in API container

# Testing
make test-basic     # Run basic tests (no dependencies)
make test           # Run all available tests
make test-docker    # Run tests in Docker
make test-full      # Run pytest with coverage

# Development
make dev            # Run development server
make lint           # Run code linting
make format         # Format code

Local Installation

Create a virtual environment:

python3 -m venv venv

Activate the virtual environment:

# On Windows
venv\Scripts\activate
# On macOS/Linux
source venv/bin/activate

Install the required packages:

pip install -r requirements.txt

Create a .env file in the root directory and set the required environment variables:

cp .env.example .env

Update the .env file with your PostgreSQL database credentials and other configurations.
Install the database migrations:

alembic upgrade head

Run the seeders to populate the database with currency data:

python app/db/seed.py

Start the application:

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Access the API documentation:

http://localhost:8000/docs

# or
http://localhost:8000/redoc

Usage

API Endpoints

Get Currency List:

GET /api/currencies
Returns a list of all available currencies.

Response:

{
    "success": true,
    "data": [
        {
        "id": 1,
        "name": "Euro",
        "name_plural": "euros",
        "code": "EUR",
        "symbol": "€",
        "decimal_digits": 2,
        "icon": null,
        "created_at": "2025-04-12T16:27:05.360172+01:00",
        "updated_at": "2025-04-12T16:27:05.360172+01:00"
        },
        {
        "id": 2,
        "name": "United Arab Emirates Dirham",
        "name_plural": "UAE dirhams",
        "code": "AED",
        "symbol": "AED",
        "decimal_digits": 2,
        "icon": null,
        "created_at": "2025-04-12T16:27:05.360172+01:00",
        "updated_at": "2025-04-12T16:27:05.360172+01:00"
        },
        ...
    }
]

Get Exchange Rate:

GET /api/rates?from=USD&to=EUR
Query parameters:
- from: The base currency code (e.g., USD).
- to: The target currency code (e.g., EUR).
- amount: The amount to convert (optional, default is 1).

Returns the exchange rate from one currency to another.

Response:

{
    "success": true,
    "data": {
        "id": 31329,
        "base_currency_id": 9,
        "target_currency_id": 1,
        "rate": 0.88019,
        "source": "trading-economics",
        "created_at": "2025-04-13T12:04:27.949186+01:00",
        "base_currency": {
        "id": 9,
        "name": "US Dollar",
        "name_plural": "US dollars",
        "code": "USD",
        "symbol": "$",
        "decimal_digits": 2,
        "icon": null,
        "created_at": "2025-04-12T16:27:05.360172+01:00",
        "updated_at": "2025-04-12T16:27:05.360172+01:00"
        },
        "target_currency": {
        "id": 1,
        "name": "Euro",
        "name_plural": "euros",
        "code": "EUR",
        "symbol": "€",
        "decimal_digits": 2,
        "icon": null,
        "created_at": "2025-04-12T16:27:05.360172+01:00",
        "updated_at": "2025-04-12T16:27:05.360172+01:00"
        },
        "amount": null,
        "converted_amount": null
    },
    "message": "Exchange rate retrieved successfully."
}

Get Exchange Rate History:

GET /api/rates/history?from=USD&to=EUR&from_date=2025-04-01&to_date=2025-04-10
Query parameters:
- from: The base currency code (e.g., USD).
- to: The target currency code (e.g., EUR).
- from_date: The start date for the history (format: YYYY-MM-DD).
- to_date: The end date for the history (format: YYYY-MM-DD).

Returns the historical exchange rates between two currencies.

Response:

{
    "success": true,
    "data": {
        "base": "USD",
        "target": "EUR",
        "rates": [
            {
                "id": 129,
                "base_currency_id": 9,
                "target_currency_id": 1,
                "rate": 0.88019,
                "source": "trading-economics",
                "created_at": "2025-04-12T23:46:41.554303+01:00"
            },
            {
                "id": 15537,
                "base_currency_id": 9,
                "target_currency_id": 1,
                "rate": 0.88019,
                "source": "trading-economics",
                "created_at": "2025-04-13T11:34:08.462136+01:00"
            },
            {
                "id": 31329,
                "base_currency_id": 9,
                "target_currency_id": 1,
                "rate": 0.88019,
                "source": "trading-economics",
                "created_at": "2025-04-13T12:04:27.949186+01:00"
            },
            ...
        ]
    },
    "message": "Exchange rate history retrieved successfully."
}

Background Tasks

Koel uses Celery to run background tasks for scraping data from multiple sources. The tasks are defined in the app/tasks/celery_app.py directory. You can run the Celery worker using the following command:

celery -A app.tasks.celery_app worker --loglevel=info

This will start the Celery worker and listen for tasks to execute. You can also run the Celery beat scheduler to schedule periodic tasks:

celery -A app.tasks.celery_app beat --loglevel=info

Caching

Koel uses Redis for caching the exchange rates to improve performance and run job and tasks. You can configure the caching settings in the .env file. By default, Koel uses a Redis instance running on localhost:6379. You can change the Redis URL in the .env file:

CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
REDIS_PASSWORD=

Database

Koel uses PostgreSQL as the database backend. You can configure the database settings in the .env file. By default, Koel uses a PostgreSQL instance running on localhost:5432. You can change the database URL in the .env file:

API_VERSION=0.1.0
API_TITLE=Koel Exchange Rate API
APP_PORT=8000

DB_CONNECTION=postgresql
DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=
DB_NAME=koel

# Set to "true" to seed the database on initialization
SEED_DB=true

Sources

Koel scrapes data from multiple sources to provide accurate and up-to-date exchange rates. The sources are defined in the app/scraping/sources directory. You can add or modify the sources by creating new classes that extend the BaseScraper class.

Supported Sources

Source Name	Description	Source URL	Pair Type
Trading Economics	Provides real-time exchange rates and economic data.	tradingeconomics.com	Multi Pair
Exchange Rates Org Uk	Provides exchange rates and currency conversion data.	exchangerates.org.uk	Multi Pair
Currency Converter Org Uk	Provides currency conversion data and exchange rates.	currencyconverter.co.uk	Multi Pair
X-Rates	Provides exchange rates and currency conversion data.	x-rates.com	Multi Pair
Forbes	Provides exchange rates and financial news.	forbes.com	Single Pair
Hexa Rates	Provides exchange rates and currency conversion data.	hexarate.paikama.co	Single Pair
FxEmpire	Provides exchange rates and financial news.	fxempire.com	Single Pair
Oanda	Provides exchange rates and financial data.	oanda.com	Single Pair
Wise	Provides exchange rates and currency conversion data.	wise.com	Single Pair
Xe	Provides exchange rates and currency conversion data.	xe.com	Single Pair

Contributing

We welcome contributions to Koel! If you would like to contribute, please follow these steps:

Fork the repository.
Create a new branch for your feature or bug fix:

git checkout -b feature/your-feature-name

Make your changes and commit them:

git commit -m "Add your feature or fix"

Push your changes to your forked repository:

git push origin feature/your-feature-name

Create a pull request to the main repository.
Describe your changes and why they should be merged.
Wait for feedback and make any necessary changes.
Once approved, your changes will be merged into the main repository.

License

Koel is licensed under the MIT License. Feel free to use, modify, and distribute this software as per the terms of the license.

Acknowledgments

FastAPI - The web framework used for building the API.
Celery - The task queue used for background tasks.
Redis - The in-memory data structure store used for caching and job tracking.
PostgreSQL - The database used for storing exchange rates and currency data.
Alembic - The database migration tool used for managing schema changes.
Docker - The containerization platform used for deploying Koel.
Docker Compose - The tool used for defining and running multi-container Docker applications.
Pydantic - The data validation and settings management library used for defining data models.
SQLAlchemy - The SQL toolkit and Object-Relational Mapping (ORM) library used for database interactions.
Beautiful Soup - The library used for web scraping and parsing HTML.
Requests - The library used for making HTTP requests.

👥 Authors

Endurance - Github - Twitter - LinkedIn

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
alembic		alembic
app		app
docs		docs
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
IDE_SETUP.md		IDE_SETUP.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
TESTING.md		TESTING.md
alembic.ini		alembic.ini
check_environment.py		check_environment.py
docker-compose.yml		docker-compose.yml
docker-entrypoint.sh		docker-entrypoint.sh
pytest.ini		pytest.ini
requirements.txt		requirements.txt
run_tests.py		run_tests.py
simple_test.py		simple_test.py
test_local.py		test_local.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Table of Contents

About Koel

Features

Getting Started

Prerequisites

Clone the Repository

Installation

Using Docker Compose

Using Makefile Commands

Local Installation

Usage

API Endpoints

Background Tasks

Caching

Database

Sources

Supported Sources

Contributing

License

Acknowledgments

👥 Authors

About

Uh oh!

Releases

Packages

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

Table of Contents

About Koel

Features

Getting Started

Prerequisites

Clone the Repository

Installation

Using Docker Compose

Using Makefile Commands

Local Installation

Usage

API Endpoints

Background Tasks

Caching

Database

Sources

Supported Sources

Contributing

License

Acknowledgments

👥 Authors

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages