Skip to content

Multi Repository Support

Jump to bottom
github-actions[bot] edited this page Mar 14, 2026 · 1 revision

Multi-Repository and Grouped Wiki Support

wikigen supports both single-repository wiki generation and multi-repository wiki generation, where multiple repositories can be grouped into a single integrated wiki. This page explains the architecture, configuration, and behavior of grouped wiki support, including how repositories are combined, cross-repository documentation is generated, and services interact in multi-repo projects.

Overview of Multi-Repository Support

wikigen distinguishes between two repository configuration modes:

  1. Standalone Mode: Each repository generates its own separate wiki independently
  2. Grouped Mode: Multiple repositories are merged into a single wiki with cross-repository documentation

The choice between these modes is determined entirely by the format used in repos.txt or CLI arguments. A single project can use either format, but not both simultaneously.

Sources: main.go:668-688, README.md:95-108

Repos.txt Configuration Format

Standalone Mode

In standalone mode, each repository generates its own independent wiki:

owner/repo1
owner/repo2
owner/repo3

Each repository listed without a project prefix creates a separate wiki directory in the output folder:

  • wiki-output/repo1/
  • wiki-output/repo2/
  • wiki-output/repo3/

Grouped Mode

In grouped mode, multiple repositories are prefixed with a project name and colon, merging them into a single wiki:

myproject:owner/frontend-repo
myproject:owner/backend-repo
myproject:owner/shared-lib

All repositories under the same project name (myproject) are combined into a single wiki directory:

  • wiki-output/myproject/

A single repos.txt file can contain both standalone and grouped repositories:

# Standalone repositories
owner/repo1
owner/repo2

# Grouped project
api-gateway:owner/gateway
api-gateway:owner/auth-service
api-gateway:owner/common-lib

# Another grouped project
frontend:owner/web-app
frontend:owner/ui-components

Sources: main.go:668-688, README.md:95-108

Repository Parsing and Task Building

Parsing Logic

The parseRepoList function distinguishes between standalone and grouped repositories by checking for a colon separator:

func parseRepoList(lines []string) (standalone []RepoEntry, groups map[string][]string) {
	groups = make(map[string][]string)
	for _, line := range lines {
		line = strings.TrimSpace(line)
		if line == "" || strings.HasPrefix(line, "#") {
			continue
		}
		if before, after, ok := strings.Cut(line, ":"); ok && !strings.Contains(before, "/") {
			// project:owner/repo format
			groups[before] = append(groups[before], after)
		} else {
			// standalone owner/repo format
			standalone = append(standalone, RepoEntry{Repo: line})
		}
	}
	return
}

The function returns two structures:

  • standalone: Array of individual repositories with empty Project field
  • groups: Map where keys are project names and values are arrays of repository identifiers

The key condition !strings.Contains(before, "/") ensures that URLs with colons (such as SSH URLs) are not misinterpreted as grouped format.

Sources: main.go:672-687

Task Building

After parsing, the main function builds a unified task list that treats both standalone and grouped repositories identically:

type task struct {
	name  string
	repos []string
}
var tasks []task

for _, entry := range standalone {
	parts := strings.Split(entry.Repo, "/")
	name := entry.Repo
	if len(parts) >= 2 {
		name = parts[len(parts)-1]
	}
	tasks = append(tasks, task{name: name, repos: []string{entry.Repo}})
}
for project, repoList := range groups {
	tasks = append(tasks, task{name: project, repos: repoList})
}
  • Standalone repositories: Each becomes a task with name derived from the repository name and repos containing a single repository
  • Grouped repositories: Each project becomes a single task with name as the project identifier and repos containing all grouped repositories

This unified task model means that grouped wikis and standalone wikis follow the same execution path.

Sources: main.go:959-975

Multi-Repository Wiki Generation Pipeline

Repository Cloning and Validation

When a grouped wiki task executes, all repositories in the group are cloned to the same clone directory before analysis begins:

var repoDirs []string
for _, repo := range repos {
	repoURL := repo
	if !strings.HasPrefix(repo, "http") {
		repoURL = fmt.Sprintf("https://github.com/%s", repo)
	}
	// Parse owner and repo name...
	repoDir, _ := filepath.Abs(filepath.Join(cloneDir, fmt.Sprintf("%s_%s", owner, repoName)))
	if err := gitClone(repoURL, token, repoDir); err != nil {
		return result, fmt.Errorf("clone %s: %w", repo, err)
	}
	repoDirs = append(repoDirs, repoDir)
}

Each repository is cloned to a directory named {owner}_{reponame} within the clone directory. For example, given repositories frontend:acme/web-app and frontend:acme/api-server:

  • Cloned to: .repos/acme_web-app/
  • Cloned to: .repos/acme_api-server/

All cloned directories are collected into a repoDirs array for passing to Claude.

Sources: main.go:488-515

Claude Code Integration

All cloned repositories are passed simultaneously to Claude Code via the --add-dir flag:

func claudeCall(claudePath, model string, repoDirs []string, systemPrompt, prompt, workDir string) (string, error) {
	args := []string{"-p", "--output-format", "text", "--dangerously-skip-permissions"}
	if model != "" {
		args = append(args, "--model", model)
	}
	for _, dir := range repoDirs {
		args = append(args, "--add-dir", dir)
	}
	// ... execute claude command
}

This allows Claude Code to access all repository source code simultaneously using its native tools (Read, Grep, Glob, Bash). For grouped wikis with N repositories, N --add-dir flags are passed.

Sources: main.go:116-143

Cross-Repository Documentation Generation

Structure Prompt Guidance

The structure determination prompt includes explicit rules for multi-repository projects:

## Rules for Multi-Repository Projects
- When multiple repositories form one project, create CROSS-REPOSITORY documentation
- Show how repositories interact with each other (e.g., frontend calls backend API)
- Create architecture pages that span all repositories
- Individual repository details should still get their own focused pages
- Clearly indicate which repository each page primarily covers

These rules are included in the prompt to Claude, guiding the XML structure generation to create pages that:

  • Document inter-service communication
  • Explain cross-repository architectural patterns
  • Identify dependencies between repositories
  • Create both integration-focused and repository-focused documentation

Sources: main.go:225-230

Example Multi-Repository Structure

For a grouped wiki like:

api-platform:owner/gateway
api-platform:owner/auth-service
api-platform:owner/database-service

The generated wiki structure might include:

  • System Architecture — Cross-repository architecture diagram showing service interactions
  • API Specification — Unified API documentation with gateway routing and service endpoints
  • Authentication Flow — Cross-service authentication chain from gateway through auth-service
  • Database Interactions — Data flow between database-service and other services
  • Gateway Implementation — gateway-specific documentation (repository-focused)
  • Auth Service Design — auth-service specific design (repository-focused)
  • Database Service API — database-service specific details (repository-focused)

This mixed approach ensures both system-wide understanding and individual component documentation.

Page Generation with Multiple Repositories

During page generation, the pagePrompt function receives all repositories in the group:

func pagePrompt(page WikiPage, allPages []WikiPage, projectName string, repos []string, language string) string {
	// ...
	return fmt.Sprintf(`You are an expert technical writer creating a wiki page.

Project: %s
Repositories: %s

You have full access to ALL repository source code via the tools available to you.
`, projectName, strings.Join(repos, ", "), ...)
}

For each page being generated in a grouped wiki, Claude receives:

  • projectName: The group name (e.g., "api-platform")
  • repos: Array of all repositories in the group (e.g., ["owner/gateway", "owner/auth-service", "owner/database-service"])

This context allows Claude to generate page content that appropriately considers all repositories, linking across them when relevant.

Sources: main.go:275-362

Output Structure for Multi-Repository Wikis

Directory Organization

For a grouped wiki, all output is placed in a single project directory:

wiki-output/{projectname}/
  ├── Home.md              # Landing page listing all repositories
  ├── _Sidebar.md          # Navigation sidebar with all pages
  ├── System-Architecture.md     # Cross-repository architecture
  ├── API-Specification.md       # Unified API docs
  ├── {Cross-Repo-Page}.md       # Other cross-repo pages
  ├── {Repo1-Specific-Page}.md   # Repository-specific pages
  ├── {Repo2-Specific-Page}.md
  └── _errors.log          # Error log (created only if errors occurred)

Home Page Repository Listing

The Home.md file generated for grouped wikis includes a repositories section:

if len(repos) > 1 {
	home.WriteString("## Repositories\n\n")
	for _, r := range repos {
		home.WriteString(fmt.Sprintf("- %s\n", r))
	}
	home.WriteString("\n")
}

This explicitly lists all repositories in the grouped wiki, making their relationships clear to readers.

Sources: main.go:638-665

Parallelism in Multi-Repository Generation

wikigen supports two levels of parallelism that are orthogonal to repository grouping:

Repository-Level Parallelism (-p flag)

The -p flag controls how many wiki generation tasks run in parallel. This applies equally to standalone and grouped wikis:

sem := make(chan struct{}, parallel)
var wg sync.WaitGroup

for _, t := range tasks {
	wg.Add(1)
	sem <- struct{}{}
	go func(t task) {
		defer wg.Done()
		defer func() { <-sem }()

		result, err := generateWiki(claudePath, model, t.name, t.repos, ...)
		// ... process result
	}(t)
}

For example, with -p 2 and three tasks (two standalone repos + one grouped project), at most two tasks execute concurrently.

Page-Level Parallelism (-pp flag)

The -pp flag controls how many pages within a single wiki are generated in parallel. This applies to both standalone and grouped wikis:

pageSem := make(chan struct{}, pageParallel)
var pageWg sync.WaitGroup

for i := range allPages {
	pageWg.Add(1)
	pageSem <- struct{}{}
	go func(idx int) {
		// Generate single page
		_, err := claudeCall(claudePath, model, repoDirs, "", pagePrompt(...), wikiDir)
	}(i)
}
pageWg.Wait()

For a grouped wiki with 20 pages and -pp 5, at most 5 pages generate concurrently, with Claude receiving all repositories on each claudeCall.

Sources: main.go:563-615, main.go:1005-1035

Input Validation for Multi-Repository Projects

All repositories in both standalone and grouped configurations are validated using the same function:

var validRepoPattern = regexp.MustCompile(`^[a-zA-Z0-9._-]+/[a-zA-Z0-9._-]+$`)

func validateRepo(repo string) error {
	if !validRepoPattern.MatchString(repo) {
		return fmt.Errorf("invalid repo format: %q (expected owner/repo)", repo)
	}
	if strings.Contains(repo, "..") {
		return fmt.Errorf("path traversal detected in repo: %q", repo)
	}
	if strings.ContainsAny(repo, ";&|`$(){}[]!~") {
		return fmt.Errorf("invalid characters in repo: %q", repo)
	}
	return nil
}

Before any cloning occurs, all repositories are validated:

  • Must match owner/repo format (only alphanumeric, dots, underscores, hyphens)
  • Must not contain path traversal sequences (..)
  • Must not contain shell injection characters

For grouped wikis, each repository in the group is validated individually.

Sources: main.go:77-92

Error Handling and Recovery

Per-Repository Error Handling

Errors in grouped wiki generation are handled at the project level. If any repository fails to clone, the entire wiki generation task fails:

for _, repo := range repos {
	// ... clone code ...
	if err := gitClone(repoURL, token, repoDir); err != nil {
		return result, fmt.Errorf("clone %s: %w", repo, err)
	}
}

This ensures that grouped wikis maintain consistency — all repositories are available for structure determination and page generation.

Page-Level Retry

Individual page generation failures are retried up to 3 times per grouped wiki:

maxRetries := 3
for attempt := 1; attempt <= maxRetries; attempt++ {
	_, err := claudeCall(claudePath, model, repoDirs, "", pagePrompt(*page, allPages, projectName, repos, language), wikiDir)
	if err != nil {
		continue
	}
	// Check if file was written...
	if readErr == nil && len(written) > 100 {
		success = true
		break
	}
}

On each retry, all repositories are re-passed to Claude via --add-dir, ensuring consistent multi-repository context.

Error Logging

Failed pages are logged to _errors.log within the grouped wiki directory:

if !success {
	appendError(wikiDir, fmt.Sprintf("Page %d/%d: %s — failed after %d attempts", idx+1, len(allPages), page.Title, maxRetries))
}

All errors for a grouped wiki (regardless of which repository's documentation failed) are accumulated in a single error log.

Sources: main.go:607-609

JSON Output for Multi-Repository Wikis

When using the -json flag, grouped wikis produce a single WikiResult object that includes all repositories:

type WikiResult struct {
	Project    string           `json:"project"`
	Repos      []string         `json:"repos"`         // All grouped repos
	OutputDir  string           `json:"output_dir"`
	Pages      []WikiPageResult `json:"pages"`
	TotalPages int              `json:"total_pages"`
	Failed     int              `json:"failed"`
	Duration   string           `json:"duration"`
	Status     string           `json:"status"`
}

Example output for a grouped wiki with 3 repositories and 20 pages:

{
  "project": "api-platform",
  "repos": ["owner/gateway", "owner/auth-service", "owner/database-service"],
  "output_dir": "/path/to/wiki-output/api-platform",
  "pages": [
    {"title": "System Architecture", "filename": "System-Architecture", "size": 5234, "status": "ok"},
    {"title": "API Specification", "filename": "API-Specification", "size": 8945, "status": "ok"},
    ...
  ],
  "total_pages": 20,
  "failed": 0,
  "duration": "2m30s",
  "status": "completed"
}

Sources: main.go:96-112

Architecture Diagram for Multi-Repository Generation

graph TD
    A["repos.txt"] -->|Parse| B{"Has Colon?"}
    B -->|No| C["Standalone Repo"]
    B -->|Yes| D["Grouped Repo"]
    C --> E["Task: name=reponame<br/>repos=[single]"]
    D --> F["Task: name=project<br/>repos=[repo1, repo2, ...]"]
    E --> G["Clone Repository"]
    F --> H["Clone All Repositories<br/>in Group"]
    G --> I["Single repoDirs entry"]
    H --> J["Multiple repoDirs entries"]
    I --> K["Structure Determination<br/>claude --add-dir"]
    J --> K
    K --> L["Parse XML<br/>Generate Pages List"]
    L --> M["Page Generation Loop<br/>Parallel -pp flag"]
    M --> N["claudeCall with<br/>ALL repoDirs"]
    N --> O["Output Wiki"]
Loading

Sources: main.go:668-688, main.go:959-975

Sequence Diagram: Multi-Repository Wiki Generation

sequenceDiagram
    participant CLI as wikigen CLI
    participant Parser as parseRepoList
    participant Task as Task Builder
    participant GenWiki as generateWiki
    participant Git as git clone
    participant Claude as Claude Code
    participant Page as Page Generator
    participant Output as File Output

    CLI->>Parser: repos.txt / CLI args
    Parser-->>Task: standalone[], groups{}
    Task->>Task: Build unified task list
    Task->>GenWiki: for each task
    GenWiki->>Git: Clone repo1
    GenWiki->>Git: Clone repo2
    GenWiki->>Git: Clone repoN
    Git-->>GenWiki: repoDirs[]
    GenWiki->>Claude: Structure prompt<br/>with --add-dir for each repo
    Claude-->>GenWiki: XML structure
    GenWiki->>Page: for each page (parallel -pp)
    Page->>Claude: Page prompt<br/>with --add-dir for each repo
    Claude-->>Page: Page content
    Page->>Output: Write .md file
    Output-->>GenWiki: File written
    GenWiki-->>CLI: WikiResult
Loading

Comparison: Standalone vs. Grouped Wikis

graph TD
    A["Input"] -->|Standalone<br/>owner/repo1<br/>owner/repo2| B["Task 1<br/>repos=[repo1]"]
    A -->|Grouped<br/>project:owner/repo1<br/>project:owner/repo2| C["Task 1<br/>repos=[repo1, repo2]"]

    B --> D["Clone repo1<br/>repoDirs=[dir1]"]
    C --> E["Clone repo1, repo2<br/>repoDirs=[dir1, dir2]"]

    D --> F["claude --add-dir dir1"]
    E --> G["claude --add-dir dir1<br/>--add-dir dir2"]

    F --> H["Single-repo context<br/>Structure analysis"]
    G --> I["Multi-repo context<br/>Cross-repo analysis"]

    H --> J["wiki-output/repo1/"]
    I --> K["wiki-output/project/"]
Loading

CLI Examples for Multi-Repository Wikis

Command-line with grouped repositories:

# Generate grouped wiki from command line
./wikigen -r "api:owner/gateway,api:owner/auth,api:owner/db" -lang en

# With parallelism
./wikigen -r "api:owner/gateway,api:owner/auth" -p 1 -pp 5

# Dry run (structure only)
./wikigen -dry-run -f repos.txt

# JSON output
./wikigen -json -f repos.txt | jq '.[] | select(.project=="api") | .repos'

Configuration via environment variables:

# .env file
WIKI_OUTPUT_DIR=./wiki-output
WIKI_CLONE_DIR=./.repos
WIKI_PARALLEL=1
WIKI_PAGE_PARALLEL=3
WIKI_LANGUAGE=en

Sources: README.md:64-93

Best Practices for Multi-Repository Wikis

Repository Naming

Use clear, descriptive project names in repos.txt:

# Clear project grouping
search-platform:company/search-engine
search-platform:company/indexer
search-platform:company/analytics

# Descriptive standalone repos
company/standalone-cli
company/internal-tool

Repository Organization

Group repositories that have significant interdependencies or form a cohesive system. Avoid grouping unrelated repositories into a single wiki.

Good grouping:

  • Frontend, Backend, Shared libraries for one product
  • API Gateway, Auth Service, Database Service for one platform
  • Web App, Mobile App, Shared SDK for one application

Poor grouping:

  • Random collection of unrelated tools
  • Repositories that don't interact with each other
  • Projects with completely different documentation needs

Cross-Repository Links

For grouped wikis, leverage cross-repository links in generated documentation. Claude will naturally identify and document:

  • How frontend repositories call backend APIs
  • How services share common libraries
  • Data flow between components
  • Dependency relationships

This is automatic and requires no special configuration.

Related Pages

Home

Clone this wiki locally