Module system

[pditommaso]

Summary

Full implementation of the Nextflow Module System, enabling first-class support for installing, managing, publishing, and running modules from a centralized registry.

Main capabilities

CLI commands (nextflow module <subcommand>)

• module install — Download and install a module from the registry to the local modules/ directory with checksum validation and integrity tracking. Supports -version and -force flags.
• module run — Execute a module directly from the registry without pre-installing. Auto-installs if needed, then delegates to the standard run command.
• module list — List all locally installed modules with version and integrity status (OK / MODIFIED / LOCAL / CORRUPTED). Supports table or JSON output.
• module remove — Remove installed modules with integrity checks to prevent accidental data loss. Supports -force and -keep-files flags.
• module search — Query the registry for modules matching a search term. Returns formatted results or JSON with metadata and relevance scores.
• module info — Display detailed module metadata including description, authors, tools, input/output parameter schema, and usage command templates. Smart parameter inference for nf-core modules.
• module publish — Publish a local module to the registry. Validates structure (requires main.nf, meta.yml, README.md), enforces size limits, bundles as tar.gz, computes checksum, and uploads via authenticated API. Supports -dry-run.

Registry client (ModuleRegistryClient)

REST API client with built-in retry logic and multi-registry fallback:

• Fetch module metadata and specific release details
• Download module bundles with SHA-256 integrity validation
• Search modules by keyword
• Publish module bundles (authenticated via NXF_REGISTRY_TOKEN or config)

Local storage management (ModuleStorage)

File-based storage in modules/scope/name/ with .module-info marker files:

• Install/extract downloaded bundles with checksum tracking
• Detect local modifications via integrity checking
• Create tar.gz bundles for publishing (excluding internal metadata)
• Atomic operations with cleanup on failure

Module resolution (ModuleResolver)

Coordinates between registry and local storage:

• Resolve module references to local paths with auto-install
• Version validation and auto-upgrade on mismatch
• Integrity warnings for locally modified modules

DSL integration

New syntax for remote module inclusion:

include { myWorkflow } from 'scope/name'
include { myProcess } from 'scope/name@1.2.0'

Pluggable resolution via RemoteModuleResolver SPI, allowing third-party registries.

Configuration (registry scope)

registry {
    url = 'https://registry.nextflow.io/api'
    apiKey = '...'  // or NXF_REGISTRY_TOKEN env var
}

Supports multiple registries with fallback.

Data models

• ModuleReference — Parses and validates scope/name format
• ModuleSpec — YAML parser for meta.yml with validation (name, version, description, license)
• InstalledModule — Tracks installed state, version, checksum, and integrity status
• ModuleChecksum — SHA-256 directory checksums for integrity verification

Security

• Archive integrity validation and zip-slip prevention
• Bundle size limits (1MB)
• Checksum verification on download

Documentation and design

• ADR documenting the full specification (adr/20251114-module-system.md)
• CLI reference docs (docs/cli.md, docs/reference/cli.md)
• Module user guide (docs/module.md)
• Implementation specs and registry API contract (specs/251117-module-system/)

Testing

Comprehensive Spock test coverage for all components (~4,000 lines of tests):

• All 7 CLI subcommands
• Registry client, storage, resolver, checksum, spec parsing
• Module reference validation and include definition integration

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	34426ea
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69b83bad90ac8f00084d7c01

[bentsherman]

The module spec schema will be defined in the nextflow-io/schemas repo. Here is the PR for it: nextflow-io/schemas#10

Comments resolved

[bentsherman]

This will probably not work for the language server, since the language server can manage multiple workspaces at once. I will need to pass a workspace root into this visitor.

I will look into this separately, no need to block this PR

[bentsherman]

@christopher-hakkaart we should follow up on the module docs later. It seems like we are duplicating some info between the CLI page and Modules page, could likely be improved

[christopher-hakkaart]

Thanks @bentsherman - will check out the docs today