Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
238 lines (165 loc) · 4.91 KB

DEVELOPMENT.md

File metadata and controls

238 lines (165 loc) · 4.91 KB

Development Guide

Repository Structure

This repository contains multiple independent script collections, each in its own directory. Each directory is designed to be used independently with its own virtual environment.

Directory-specific Dependencies

Each script directory (e.g., FmiAPI/, AQBurk2IoTHub/, etc.) manages its own dependencies using:

  • pyproject.toml - Modern Python project configuration with metadata and dependency specifications
  • requirements.txt - Lock file with pinned versions (auto-generated, never edit manually)

Why This Structure?

  • Isolation: Scripts run in different environments (dev, production, different servers)
  • Minimal Dependencies: Each environment only installs what it needs
  • Reproducibility: Lock files ensure consistent installations across environments
  • Independence: Each directory can be deployed separately

Setting Up a New Script Directory

Option 1: Using uv (Recommended)

uv is a fast, modern Python package manager written in Rust.

  1. Create pyproject.toml in your directory:
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "your-script-name"
version = "0.1.0"
description = "Description of your script"
requires-python = ">=3.13"
dependencies = [
    "httpx",
    "pandas",
    # Add your dependencies here
]

[project.optional-dependencies]
dev = [
    "pytest",
    "ruff",
]
  1. Generate lock file:
cd YourDirectory/
uv pip compile pyproject.toml -o requirements.txt
  1. Create virtual environment and install:
uv venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
uv pip sync requirements.txt
  1. If using fvhdms shared module:
uv pip install -e ../fvhdms

Option 2: Using pip-tools

cd YourDirectory/
pip install pip-tools
pip-compile pyproject.toml -o requirements.txt
pip-sync requirements.txt

Option 3: Traditional pip (Not Recommended)

If you only have requirements.txt without version pinning:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Updating Dependencies

Adding a New Dependency

  1. Edit pyproject.toml and add the package to dependencies:
dependencies = [
    "existing-package",
    "new-package",  # Add here
]
  1. Regenerate lock file:
uv pip compile pyproject.toml -o requirements.txt
  1. Update your environment:
uv pip sync requirements.txt

Upgrading Dependencies

To upgrade all dependencies to their latest compatible versions:

uv pip compile --upgrade pyproject.toml -o requirements.txt
uv pip sync requirements.txt

Common Patterns

Shared fvhdms Module

Many scripts use the shared fvhdms module located in the repository root. Install it in editable mode:

pip install -e ../fvhdms
# or with uv:
uv pip install -e ../fvhdms

Environment Variables

Some scripts require environment variables or .env files:

# Copy example config
cp config.ini.example config.ini

# Edit with your settings
nano config.ini

Running Scripts in Production

In production environments, clone the repo and set up only the needed directory:

git clone <repo-url>
cd DataManagementScripts/FmiAPI/
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e ../fvhdms  # if needed
python fmiapi.py --help

Tools Used

uv - Fast Python Package Manager

ruff - Fast Python Linter & Formatter

Configured in root pyproject.toml:

# Lint all files
ruff check .

# Format code
ruff format .

# Auto-fix issues
ruff check --fix .

pre-commit - Git Hooks

See .pre-commit-config.yaml for automated checks before commits.

Migrating Existing Directories

To migrate a directory from simple requirements.txt to pyproject.toml:

  1. Analyze current dependencies:
cd YourDirectory/
pip list --format=freeze > current-deps.txt

  1. Create pyproject.toml with direct dependencies only (not transitive)

  2. Generate new lock file:

uv pip compile pyproject.toml -o requirements.txt
  1. Test in fresh environment:
uv venv test-env
source test-env/bin/activate
uv pip sync requirements.txt
python your_script.py --help
  1. Clean up:
deactivate
rm -rf test-env current-deps.txt

Example: FmiAPI Directory

See FmiAPI/ as a reference implementation:

  • ✅ Modern pyproject.toml with metadata
  • ✅ Auto-generated requirements.txt lock file
  • ✅ Updated README.md with installation instructions
  • ✅ Optional dev dependencies for testing

Questions?

  • Check existing directories like FmiAPI/ for examples
  • Review this guide for best practices
  • Tools documentation: uv, pip-tools, setuptools