Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,18 @@ jobs:
- name: Run ruff
run: python -m ruff check src/ tests/

- name: Run tests
run: python -m pytest tests/ -v --tb=short
- name: Run tests (with coverage on ubuntu x py3.11)
run: |
if [ "${{ matrix.os }}" = "ubuntu-latest" ] && [ "${{ matrix.python-version }}" = "3.11" ]; then
python -m pytest tests/ -v --tb=short --cov --cov-report=term-missing --cov-report=xml
else
python -m pytest tests/ -v --tb=short
fi
shell: bash

- name: Upload coverage artifact
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
uses: actions/upload-artifact@v4
with:
name: coverage-attune-author
path: coverage.xml
224 changes: 224 additions & 0 deletions docs/specs/regen-pipeline/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,224 @@
# Spec: Regen Pipeline — Design

## Phase 2: Design

**Status**: in-review

---

### Architecture

```
┌──────────────────────────────────────────────────────────────────┐
│ attune-gui React UI │
│ CorpusSetup → path input + Browse button + Load button │
│ App → DashboardSummaryBar + "Regen all stale" button │
│ StaleBadge → per-row refresh (unchanged) │
└────────────────────────┬─────────────────────────────────────────┘
│ HTTP / WebSocket
┌────────────────────────▼─────────────────────────────────────────┐
│ attune-gui FastAPI sidecar │
│ GET /api/config ← read current corpus root │
│ POST /api/config ← set corpus root + reload │
│ GET /api/browse/directory ← native dir picker (macOS) │
│ POST /api/templates/refresh-all ← bulk regen, returns job IDs │
│ (existing endpoints unchanged) │
└──────────┬──────────────────────────────┬────────────────────────┘
│ library call │ library call
┌──────────▼───────────┐ ┌──────────────▼────────────────────────┐
│ attune-rag │ │ attune-author │
│ DirectoryCorpus │ │ regen_template(path, corpus_root) │
│ (unchanged) │ │ _regen → polish + summary + write │
└──────────────────────┘ └───────────────────────────────────────┘
```

---

### API changes

#### New: `GET /api/config`

```
Response 200:
{
"corpus_root": "/abs/path/to/templates" | null
}
```

Returns the currently loaded corpus root. `null` if no corpus is loaded.

---

#### New: `POST /api/config`

```
Request: { "corpus_root": "/abs/path/to/templates" }
Response 200:
{
"corpus_root": "/abs/path/to/templates",
"template_count": 26
}
```

Validates the path exists, calls `load_corpus(corpus_root)`, returns the count.
Returns 422 if the path does not exist or is not a directory.

---

#### New: `GET /api/browse/directory`

Opens a native macOS Finder directory-picker dialog (via `tkinter.filedialog`) in a
thread, waits for the user to select a folder, and returns the chosen path.

```
Response 200: { "path": "/abs/path/to/templates" }
Response 204: {} ← user cancelled the dialog
```

This endpoint blocks until the dialog closes (typically < 5s). The frontend fires it
on Browse button click and populates the path input with the result.

---

#### New: `POST /api/templates/refresh-all`

Creates refresh jobs for every template whose current staleness is `"stale"` or
`"warning"`. Returns all job IDs immediately (202); the client connects to each WS
individually using the existing `WS /ws/refresh/{job_id}` endpoint.

```
Response 202:
{
"jobs": [
{ "job_id": "uuid", "path": "concepts/auth.md", "status": "pending" },
...
],
"total": 8
}
```

---

### attune-author: `regen_template` signature change

```python
def regen_template(
template_path: str,
corpus_root: str | Path | None = None,
) -> None:
```

Resolution order for `corpus_root`:
1. Explicit parameter
2. `ATTUNE_CORPUS_ROOT` environment variable
3. `.env` file in the current working directory (`python-dotenv` loads it at call time)
4. `RuntimeError` — cannot proceed

`_regen` implementation flow:

```
1. _resolve_corpus_root(corpus_root) → Path
2. load .env (python-dotenv) if present
3. check ANTHROPIC_API_KEY — raise RuntimeError if missing
4. full_path = corpus_root / template_path — raise FileNotFoundError if missing
5. post = frontmatter.load(full_path)
6. client = anthropic.Anthropic()
7. polish_response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=4096,
system=[{"type":"text","text":SYSTEM_POLISH,"cache_control":{"type":"ephemeral"}}],
messages=[{"role":"user","content": post.content}]
)
8. improved = polish_response.content[0].text
9. summary_response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=128,
messages=[{"role":"user","content": f"One sentence summary:\n\n{improved}"}]
)
10. post.content = improved
11. post.metadata["summary"] = summary_response.content[0].text.strip()
12. atomic_write(full_path, frontmatter.dumps(post)) ← temp → rename
13. _patch_summaries_json(corpus_root, template_path, post.metadata["summary"])
```

`atomic_write`: write to `full_path.with_suffix(".tmp")`, then `os.replace()`.
`_patch_summaries_json`: load `corpus_root/summaries.json` if present, update the
matching key, write back. No-op if file absent.

---

### Data model changes

**New: `ConfigState`** (in `attune_gui/config.py`, module-level singleton)

```python
class ConfigState(BaseModel):
corpus_root: Path | None = None
```

Replaces the current implicit `_corpus` in `corpus_adapter.py`. Both modules share it.

**No changes** to `TemplateEntry`, `JobState`, or `summaries.json` schema.

---

### UI/UX

#### Corpus setup screen (shown when `corpus_root is None`)

```
┌──────────────────────────────────────────────────────────────┐
│ Attune Template Dashboard │
│ │
│ No corpus loaded. │
│ ┌─────────────────────────────────────┐ [Browse] [Load] │
│ │ /path/to/templates │ │
│ └─────────────────────────────────────┘ │
│ ← inline error if path invalid │
└──────────────────────────────────────────────────────────────┘
```

- **Browse** fires `GET /api/browse/directory`; populates the text input with returned path.
Disabled + shows spinner while the dialog is open.
- **Load** fires `POST /api/config`; on success replaces the setup screen with the
template list. On 422, shows the error message inline below the input.

#### "Regen all stale" button (shown when `summary.stale > 0 || summary.warning > 0`)

Placed in `DashboardSummaryBar`, after the counts:

```
● 3 stale · ● 5 warning · 26 total [Regen all stale]
```

Click flow:
1. Button fires `POST /api/templates/refresh-all`.
2. For each returned job, connects a WebSocket as normal.
3. Button shows "Regenerating 8…" with a running count of completed jobs.
4. Button re-enables when all jobs reach `done` or `error`.
5. Rows update badge-by-badge via existing `onDone` / `onError` logic.

---

### Cross-layer impact

| Order | Layer | Change |
|-------|-------|--------|
| 1 | attune-author | `regen_template` signature + `_regen` implementation; add `python-dotenv` dep |
| 2 | attune-gui sidecar | `config.py` module; 3 new routes; `_run_regen` passes corpus root; sidecar startup auto-loads from env |
| 3 | attune-gui UI | `CorpusSetup` component; `DashboardSummaryBar` gains Regen-all button; `App` checks corpus state on mount |

attune-rag and attune-help: no changes.

---

### Tradeoffs & alternatives

| Option | Pros | Cons | Chosen? |
|--------|------|------|---------|
| tkinter dir picker via sidecar endpoint | Native macOS dialog, no Electron | Blocks sidecar thread; must run in thread pool; won't work headless | **Yes** |
| Web File System Access API | Pure browser, no sidecar change | Returns `FileSystemDirectoryHandle`, not a path string — useless for sidecar | No |
| Startup flag only (`--corpus`) | Simplest | No in-app reconfiguration; fails the UX requirement | No |
| Bulk regen via individual POST per badge | Reuses existing flow | N clicks, no single "regen all" affordance | No |
| Bulk regen `refresh-all` endpoint | Single click, server manages job creation | Slightly more server code | **Yes** |
| python-dotenv for API key | Dev-friendly; key lives in `.env` alongside code | Adds a dep to attune-author | **Yes** |
75 changes: 75 additions & 0 deletions docs/specs/regen-pipeline/requirements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Spec: Regen Pipeline

> Extends the staleness-badge feature. Completes task 18 (smoke test) by implementing
> `attune_author.regen._regen` and wiring corpus root through the sidecar + UI.

---

## Phase 1: Requirements

**Status**: approved

### Problem statement

The staleness-badge feature (tasks 1–17) is complete. Task 18 requires `_regen` to
be callable end-to-end: clicking a stale badge in the dashboard must regenerate the
template file on disk and clear the badge to "fresh". Currently `_regen` raises
`NotImplementedError`, so the smoke test cannot run.

Additionally, the sidecar has no way to know where templates live on disk, and the
dashboard has no UI for pointing it to the right corpus root.

### Scope

**In scope:**

- Implement `attune_author.regen._regen(template_path, corpus_root)`:
- Load the existing template file
- Call Claude (Sonnet) to polish the Markdown content
- Call Claude (Haiku) to generate a fresh one-sentence summary
- Write the result back atomically (temp file → rename)
- Update the `summary` field in the file's YAML frontmatter
- Patch the matching entry in `summaries.json` (if present in corpus root)
- Add `corpus_root: str | Path | None = None` to `regen_template` public signature
(env var `ATTUNE_CORPUS_ROOT` as fallback)
- Sidecar: auto-load corpus from `ATTUNE_CORPUS_ROOT` at startup; expose
`GET /api/config` and `POST /api/config` to read/set the corpus root at runtime
- Sidecar WS handler: pass corpus root to `regen_template`
- Dashboard UI: if corpus root is not loaded on startup, show a text input + "Load"
button at the top of the template list; once set, templates appear normally
- Native directory picker (Browse button — text input + "Load" )
- Bulk regen (regenning all stale templates at once)

**Out of scope:**

- Rollback history / undo
- Embedding freshness into a separate freshness-score field (staleness is mtime-based)
- Updating `summaries.json` entries for templates other than the one being regenned

### User stories

1. As a developer, I click a stale badge and the template is polished by Claude and
saved to disk, so my corpus stays current without manual editing.
2. As a developer running the dashboard for the first time, I can type my corpus root
path in a field and click "Load" so templates appear without needing to set an env var.
3. As a developer who sets `ATTUNE_CORPUS_ROOT` before starting the sidecar, templates
load immediately on first open — no setup screen.

### Edge cases & open questions

| Question / Edge case | Resolution |
|---|---|
| Template file has no existing `summary` in frontmatter | Create the field; don't fail |
| `summaries.json` does not exist in corpus root | Skip the update; don't create the file |
| Claude API key is in the projects .env files | Fail with `RuntimeError("ANTHROPIC_API_KEY not set")`; sidecar emits error frame |
| Template file is missing from corpus root | Fail with `FileNotFoundError`; sidecar emits error frame |
| Polish call returns content that drops YAML-looking lines | Replace only `post.content`; frontmatter metadata is never touched by the LLM |
| User types a non-existent path in the corpus root field | Sidecar returns 422; UI shows inline error |
| Atomic write: process killed between temp write and rename | Temp file left behind (acceptable); original intact |

### Affected layers

- [x] attune-rag — no changes
- [x] attune-gui (sidecar + UI)
- [x] attune-author
- [ ] attune-help — no changes
48 changes: 48 additions & 0 deletions docs/specs/regen-pipeline/tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Spec: Regen Pipeline — Tasks

## Phase 3: Tasks

**Status**: complete

> Shipped: `attune-author regenerate` CLI lives in `src/attune_author/cli.py:507` (handler) with the parser registered around line 154. Core logic in `maintenance.py` and `maintenance_batch.py`. CHANGELOG documents the batch variant.

### Implementation order

| # | Task | Layer | Status | Notes |
|---|------|-------|--------|-------|
| 1 | Add `python-dotenv` to attune-author deps | attune-author | done | `pyproject.toml` required + ai extras |
| 2 | Add `_resolve_corpus_root(corpus_root)` helper | attune-author | done | param → `ATTUNE_CORPUS_ROOT` env → `.env` file → `RuntimeError` |
| 3 | Add `atomic_write(path, text)` helper | attune-author | done | Write to `path.with_suffix(".tmp")`, then `os.replace()` |
| 4 | Add `_patch_summaries_json(corpus_root, template_path, summary)` helper | attune-author | done | Load, update key, write back; no-op if file absent |
| 5 | Update `regen_template` signature to `(template_path, corpus_root=None)` | attune-author | done | Call `_resolve_corpus_root`; pass root to `_regen` |
| 6 | Implement `_regen(template_path, corpus_root)` | attune-author | done | Load .env, check API key, load template, Sonnet polish, Haiku summary, atomic write, patch summaries.json |
| 7 | Update `test_regen.py` for new signature | attune-author | done | Fix `assert_called_once_with` args |
| 8 | Add tests for `_resolve_corpus_root` | attune-author | done | param wins; falls back to env var; falls back to .env; raises when all missing |
| 9 | Add tests for `_regen` | attune-author | done | Mock `anthropic.Anthropic`; verify atomic write creates correct file; verify summaries.json patched; verify no-op when summaries.json absent |
| 10 | Add `attune_gui/config.py` — `ConfigState` singleton + `get_config()` / `set_corpus_root()` | attune-gui | done | Module-level `_config: ConfigState`; `set_corpus_root` calls `load_corpus` and stores root |
| 11 | Refactor `corpus_adapter.py` to use `config.get_config().corpus_root` | attune-gui | done | Remove module-level `_corpus`; delegate root tracking to config module |
| 12 | Add `attune_gui/routes/config.py` — `GET /api/config` and `POST /api/config` | attune-gui | done | POST validates path exists + is dir (422 otherwise); calls `set_corpus_root`; returns count |
| 13 | Add `GET /api/browse/directory` to config routes | attune-gui | done | osascript subprocess (replaced tkinter — crashes on macOS from non-main thread) |
| 14 | Wire config router into `main.py`; auto-load `ATTUNE_CORPUS_ROOT` on startup | attune-gui | done | Call `set_corpus_root` in FastAPI `lifespan` if env var is set |
| 15 | Update `_run_regen` in `ws.py` to pass `corpus_root` from config | attune-gui | done | `from attune_gui.config import get_config; root = get_config().corpus_root` |
| 16 | Add `POST /api/templates/refresh-all` to templates routes | attune-gui | done | Create jobs for all entries whose staleness is stale or warning; return 202 + job list |
| 17 | Add tests for `GET /api/config` and `POST /api/config` | attune-gui | done | GET returns null when unset; POST valid path returns count; POST missing path returns 422 |
| 18 | Add test for `GET /api/browse/directory` | attune-gui | done | Mocks osascript subprocess; asserts 200 with path and 204 on cancel |
| 19 | Add test for `POST /api/templates/refresh-all` | attune-gui | done | Mock `get_entries` with mixed staleness; assert only stale + warning get jobs; 202 response shape |
| 20 | Build `CorpusSetup` React component | attune-gui UI | done | Props: `onLoaded(corpusRoot)`; text input + Browse button + Load button; inline error on 422 |
| 21 | Update `App.jsx` — check `GET /api/config` on mount; show `CorpusSetup` when `corpus_root` is null | attune-gui UI | done | Replace template-list render with `<CorpusSetup>` until corpus is set |
| 22 | Update `DashboardSummaryBar` — add "Regen all stale" button | attune-gui UI | done | Shown when `stale + warning > 0`; fires `POST /api/templates/refresh-all`; shows running count |
| 23 | Update `App.jsx` — handle bulk regen response (connect WS per job, update badge per `done`/`error`) | attune-gui UI | done | Reuse `handleDone` / `handleError`; track count of completed jobs in summary bar button |
| 24 | Manual smoke test | attune-gui UI | done | Corpus loaded, stale badge clicked → spinner → fresh, "Regen all stale" confirmed working end-to-end |

### Testing strategy

- **attune-author (tasks 7–9)**: pytest unit tests. Mock `anthropic.Anthropic` to avoid real API calls. Use `tmp_path` for file I/O assertions.
- **attune-gui sidecar (tasks 17–19)**: pytest + FastAPI `TestClient`. Mock `tkinter.filedialog`, mock `get_entries`, mock `set_corpus_root` where needed.
- **attune-gui UI (tasks 20–23)**: No automated test suite yet. Task 24 (manual smoke) is the v1 acceptance gate.

### Rollback plan

- **attune-author (tasks 1–9)**: New helpers + signature change with default param — fully backwards compatible. Revert commits.
- **attune-gui sidecar (tasks 10–19)**: `config.py` is new; routes are additive; `_run_regen` change is small. Reverting does not break the existing staleness-badge feature.
- **attune-gui UI (tasks 20–23)**: `CorpusSetup` is a new component; `App.jsx` falls back gracefully if `GET /api/config` 404s. Revert commits.
Loading
Loading