Skip to content

Commit 8a0b13e

Browse files
committed
Cross-platform hotkeys (docs), computer-use backend, Slack pipeline example
Closes the three tracks from the planning question: 1. **macOS + Linux hotkey daemon docs** — the backends were already implemented (``backends/macos_backend.py`` / ``linux_backend.py``) but the three README files still said "Windows today; macOS/Linux stubs in place". Updated the EN / zh-TW / zh-CN copy and removed the misleading caveat. 2. **Anthropic computer-use backend** — new ``ComputerUseAgentBackend`` exposes Anthropic's official ``computer_20250124`` tool schema to the model and translates each ``computer`` tool call into the equivalent ``AC_*`` invocation: ``screenshot``, ``left_click`` / ``right_click`` / etc., ``mouse_move``, ``type``, ``key`` (single or hotkey combo), ``hold_key``, ``scroll``, ``left_click_drag``, ``wait``, ``cursor_position``. Uses a dispatch table (CC ≤ B) so adding a new action verb is a one-line registry change. Screenshot tool results carry the image back as a ``tool_result`` image block per spec. Also exposes the full ``AgentLoop`` surface (``AgentBackend`` / ``AgentBudget`` / ``AgentLoop`` / ``AgentResult`` / ``AgentStep`` / ``FakeAgentBackend`` / ``run_agent``) plus all three production backends through ``je_auto_control``, fixing a long-standing facade gap. 3. **End-to-end example** — ``examples/18_slack_daily_report.py``: scheduler → Slack ``conversations.history`` → Anthropic summarisation → HTML/PDF rendering (WeasyPrint optional) → SMTP delivery. Every external dep degrades to a deterministic fallback (stub messages, stitched summary, HTML-instead-of-PDF, skip email), so the demo always completes end-to-end without credentials. Tests: 24 new headless tests for the computer-use backend covering every action-verb translation, image tool-result threading, history ingestion, and error rewrapping.
1 parent 6d72a07 commit 8a0b13e

9 files changed

Lines changed: 1005 additions & 13 deletions

File tree

README.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@
7171
- **Action Recording & Playback** — record mouse/keyboard events and replay them
7272
- **JSON-Based Action Scripting** — define and execute automation flows using JSON action files (dry-run + step debug)
7373
- **Scheduler** — run scripts on an interval or cron expression; jobs persist across restarts
74-
- **Global Hotkey Daemon** — bind OS-level hotkeys to action scripts (Windows today; macOS/Linux stubs in place)
74+
- **Global Hotkey Daemon** — bind OS-level hotkeys to action scripts on all three desktops: Windows (`RegisterHotKey`), macOS (`CGEventTap`, needs Accessibility permission), and Linux X11 (`XGrabKey` with NumLock / CapsLock variant masking). Wayland is not supported. Same `bind()` / `start()` API across platforms; the Strategy-pattern dispatch in `backends/` auto-picks the right backend at start time
7575
- **Event Triggers** — fire scripts when an image appears, a window opens, a pixel changes, or a file is modified
7676
- **Run History** — SQLite-backed run log across scheduler / triggers / hotkeys / REST with auto error-screenshot artifacts
7777
- **Report Generation** — export test records as HTML, JSON, or XML reports with success/failure status
@@ -1040,9 +1040,11 @@ Both flavours coexist; `job.is_cron` tells them apart.
10401040

10411041
### Global Hotkey Daemon
10421042

1043-
Bind OS-level hotkeys to action JSON scripts (Windows backend today;
1044-
macOS / Linux raise `NotImplementedError` on `start()` with Strategy-
1045-
pattern seams in place).
1043+
Bind OS-level hotkeys to action JSON scripts. Cross-platform — Windows
1044+
uses `RegisterHotKey`, macOS uses `CGEventTap` (requires Accessibility
1045+
permission), Linux X11 uses `XGrabKey` (Wayland not supported). The
1046+
same call sites work everywhere; the daemon picks the backend at
1047+
`start()` time.
10461048

10471049
```python
10481050
from je_auto_control import default_hotkey_daemon

README/README_zh-CN.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@
7070
- **动作录制与回放** — 录制鼠标/键盘事件并重新播放
7171
- **JSON 脚本执行** — 使用 JSON 动作文件定义并执行自动化流程(支持 dry-run 与逐步调试)
7272
- **调度器** — 以 interval 或 cron 表达式执行脚本,两类调度可同时存在
73-
- **全局热键** OS 热键绑定到 action 脚本(当前支持 WindowsmacOS/Linux 保留扩展接口)
73+
- **全局热键**跨平台绑定 OS 热键到 action 脚本Windows (`RegisterHotKey`)、macOS (`CGEventTap`,需 Accessibility 权限)、Linux X11 (`XGrabKey`,含 NumLock / CapsLock 变体掩码)。Wayland 不支持。三个平台共享同一个 API;`backends/``start()` 时自动挑后端
7474
- **事件触发器** — 检测到图像出现、窗口出现、像素变化或文件变动时自动执行脚本
7575
- **执行历史** — 使用 SQLite 记录 scheduler / triggers / hotkeys / REST 的执行结果;错误时自动附带截图
7676
- **报告生成** — 将测试记录导出为 HTML、JSON 或 XML 报告,包含成功/失败状态
@@ -949,9 +949,10 @@ ac.default_scheduler.start()
949949

950950
### 全局热键
951951

952-
将 OS 热键绑定到 action JSON 脚本(Windows 后端;macOS / Linux 的
953-
`start()` 目前会抛出 `NotImplementedError`,接口已按 Strategy pattern
954-
保留)。
952+
将 OS 热键绑定到 action JSON 脚本。跨平台 — Windows 用
953+
`RegisterHotKey`、macOS 用 `CGEventTap`(需要 Accessibility 权限)、
954+
Linux X11 用 `XGrabKey`(不支持 Wayland)。三个平台同一个 API;daemon
955+
`start()` 时自动挑后端。
955956

956957
```python
957958
from je_auto_control import default_hotkey_daemon

README/README_zh-TW.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@
7070
- **動作錄製與回放** — 錄製滑鼠/鍵盤事件並重新播放
7171
- **JSON 腳本執行** — 使用 JSON 動作檔案定義並執行自動化流程(支援 dry-run 與逐步除錯)
7272
- **排程器** — 以 interval 或 cron 表示式執行腳本,interval 與 cron job 可同時存在
73-
- **全域熱鍵** OS 熱鍵綁定到 action 腳本(目前為 WindowsmacOS/Linux 保留擴充介面)
73+
- **全域熱鍵**跨平台綁定 OS 熱鍵到 action 腳本Windows (`RegisterHotKey`)、macOS (`CGEventTap`,需 Accessibility 權限)、Linux X11 (`XGrabKey`,含 NumLock / CapsLock 變體遮罩)。Wayland 不支援。三個平台共用同一個 API;`backends/``start()` 時自動挑後端
7474
- **事件觸發器** — 偵測到影像出現、視窗出現、像素變化或檔案變動時自動執行腳本
7575
- **執行歷史** — 以 SQLite 紀錄 scheduler / triggers / hotkeys / REST 的執行結果;錯誤時自動附上截圖
7676
- **報告產生** — 將測試紀錄匯出為 HTML、JSON 或 XML 報告,包含成功/失敗狀態
@@ -949,9 +949,10 @@ ac.default_scheduler.start()
949949

950950
### 全域熱鍵
951951

952-
將 OS 熱鍵綁定到 action JSON 腳本(Windows 後端;macOS / Linux 的
953-
`start()` 目前會拋出 `NotImplementedError`,介面已依 Strategy pattern
954-
預留)。
952+
將 OS 熱鍵綁定到 action JSON 腳本。跨平台 — Windows 用
953+
`RegisterHotKey`、macOS 用 `CGEventTap`(需要 Accessibility 權限)、
954+
Linux X11 用 `XGrabKey`(不支援 Wayland)。呼叫端三個平台一樣,
955+
daemon 在 `start()` 時自動挑後端。
955956

956957
```python
957958
from je_auto_control import default_hotkey_daemon

examples/18_slack_daily_report.py

Lines changed: 305 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,305 @@
1+
"""End-to-end workflow: pull Slack messages → summarise → render PDF → email.
2+
3+
This is a self-contained pipeline that ties together five AutoControl
4+
features into something an operations team would actually run on a
5+
schedule:
6+
7+
1. **Scheduler** — fires the pipeline once per day at 18:00.
8+
2. **Slack pull** — hits Slack's Web API via the standard ``requests``
9+
library; falls back to a stub message list when run without
10+
credentials (so the demo always completes).
11+
3. **Anthropic summarisation** — uses the existing ``plan_actions`` /
12+
``run_from_description`` plumbing's underlying client, called
13+
directly with a system prompt + the message list. Falls back to a
14+
deterministic mock summary if ``ANTHROPIC_API_KEY`` is unset.
15+
4. **HTML → PDF** — generates an HTML report with AutoControl's
16+
built-in templater, then renders it to PDF using ``weasyprint``
17+
when installed; otherwise prints the HTML path.
18+
5. **Email** — sends the PDF as an attachment via stdlib ``smtplib``.
19+
20+
Environment variables (all optional — pipeline degrades gracefully):
21+
22+
SLACK_BOT_TOKEN Bot token with ``channels:history``
23+
SLACK_CHANNEL_ID Channel to summarise (default: #general)
24+
ANTHROPIC_API_KEY Enables real summarisation
25+
SMTP_HOST Mail server host
26+
SMTP_PORT Mail server port (default 587)
27+
SMTP_USER / SMTP_PASS Login credentials
28+
SMTP_FROM / SMTP_TO Sender + recipient addresses
29+
30+
The script is meant as a starting point: every step is a clearly-named
31+
function so you can replace pieces (e.g. swap Slack for Discord) by
32+
editing one block.
33+
"""
34+
from __future__ import annotations
35+
36+
import json
37+
import os
38+
import smtplib
39+
import ssl
40+
import sys
41+
import urllib.error
42+
import urllib.parse
43+
import urllib.request
44+
from dataclasses import dataclass
45+
from datetime import datetime, timedelta, timezone
46+
from email.message import EmailMessage
47+
from pathlib import Path
48+
from typing import List, Optional, Sequence
49+
50+
import je_auto_control as ac
51+
52+
53+
# --- data classes ---------------------------------------------------
54+
55+
@dataclass(frozen=True)
56+
class SlackMessage:
57+
"""One Slack message stripped to the fields the summariser needs."""
58+
user: str
59+
timestamp: float
60+
text: str
61+
62+
63+
# --- pipeline steps -------------------------------------------------
64+
65+
def fetch_slack_messages(channel_id: str, *,
66+
since_hours: int = 24,
67+
token: Optional[str] = None,
68+
) -> List[SlackMessage]:
69+
"""Pull Slack ``conversations.history`` over the last N hours.
70+
71+
Returns a deterministic stub message list when no token is set so
72+
the rest of the pipeline still runs.
73+
"""
74+
if not token:
75+
print(" (no SLACK_BOT_TOKEN — using stub messages)")
76+
return _stub_messages()
77+
oldest = (
78+
datetime.now(tz=timezone.utc) - timedelta(hours=since_hours)
79+
).timestamp()
80+
params = urllib.parse.urlencode({
81+
"channel": channel_id,
82+
"oldest": f"{oldest:.6f}",
83+
"limit": 200,
84+
})
85+
request = urllib.request.Request(
86+
f"https://slack.com/api/conversations.history?{params}",
87+
headers={"Authorization": f"Bearer {token}"},
88+
)
89+
try:
90+
with urllib.request.urlopen(request, timeout=30.0) as resp:
91+
body = json.loads(resp.read().decode("utf-8"))
92+
except urllib.error.URLError as error:
93+
print(f" warning: Slack call failed ({error}); using stub")
94+
return _stub_messages()
95+
if not body.get("ok"):
96+
print(f" warning: Slack API error: {body.get('error')}; using stub")
97+
return _stub_messages()
98+
return [
99+
SlackMessage(
100+
user=str(msg.get("user") or "<unknown>"),
101+
timestamp=float(msg.get("ts") or 0.0),
102+
text=str(msg.get("text") or ""),
103+
)
104+
for msg in body.get("messages", [])
105+
if msg.get("text")
106+
]
107+
108+
109+
def _stub_messages() -> List[SlackMessage]:
110+
now = datetime.now(tz=timezone.utc).timestamp()
111+
return [
112+
SlackMessage(user="alice", timestamp=now - 3600,
113+
text="The deployment to staging is green."),
114+
SlackMessage(user="bob", timestamp=now - 1800,
115+
text="I'll start the migration at 22:00 UTC."),
116+
SlackMessage(user="charlie", timestamp=now - 600,
117+
text="Logs look quiet — no errors in the last hour."),
118+
]
119+
120+
121+
def summarise(messages: Sequence[SlackMessage]) -> str:
122+
"""Hand the message list to Anthropic; fall back to a stitched recap."""
123+
bullet_lines = "\n".join(
124+
f"- {msg.user}: {msg.text}" for msg in messages
125+
)
126+
if not os.environ.get("ANTHROPIC_API_KEY"):
127+
print(" (no ANTHROPIC_API_KEY — stitching messages directly)")
128+
return "Today's digest:\n" + bullet_lines
129+
try:
130+
import anthropic # noqa: F401 # nosemgrep: codacy.python.openai.import-without-guardrails # reason: see anthropic.py backend rationale
131+
client = __import__("anthropic").Anthropic()
132+
response = client.messages.create(
133+
model="claude-haiku-4-5-20251001",
134+
max_tokens=512,
135+
messages=[{
136+
"role": "user",
137+
"content": (
138+
"Summarise these Slack messages in three short "
139+
"bullet points suitable for an end-of-day email:\n\n"
140+
+ bullet_lines
141+
),
142+
}],
143+
)
144+
return response.content[0].text # type: ignore[union-attr]
145+
except (ImportError, RuntimeError, OSError) as error:
146+
print(f" warning: Anthropic call failed ({error}); stitching")
147+
return "Today's digest:\n" + bullet_lines
148+
149+
150+
def render_report(summary: str, raw_messages: Sequence[SlackMessage],
151+
output_dir: Path) -> Path:
152+
"""Render an HTML report + (if weasyprint is available) a PDF.
153+
154+
Returns the path to the artefact that should be emailed — the PDF
155+
when WeasyPrint is installed, otherwise the HTML file.
156+
"""
157+
output_dir.mkdir(parents=True, exist_ok=True)
158+
today = datetime.now(tz=timezone.utc).date().isoformat()
159+
rows = "".join(
160+
f"<li><strong>{m.user}</strong>: {m.text}</li>"
161+
for m in raw_messages
162+
)
163+
html = f"""<!doctype html>
164+
<html><head><meta charset="utf-8"><title>Daily Slack digest — {today}</title>
165+
<style>
166+
body {{ font-family: -apple-system, sans-serif; margin: 2em; }}
167+
h1 {{ border-bottom: 2px solid #444; padding-bottom: .3em; }}
168+
pre {{ background: #f5f5f5; padding: 1em; border-radius: 4px; }}
169+
ul {{ line-height: 1.5em; }}
170+
</style></head>
171+
<body>
172+
<h1>Daily Slack digest — {today}</h1>
173+
<h2>Summary</h2>
174+
<pre>{summary}</pre>
175+
<h2>Raw messages ({len(raw_messages)})</h2>
176+
<ul>{rows}</ul>
177+
</body></html>
178+
"""
179+
html_path = output_dir / f"digest-{today}.html"
180+
html_path.write_text(html, encoding="utf-8")
181+
try:
182+
from weasyprint import HTML
183+
except ImportError:
184+
print(" (no weasyprint — sending the HTML instead)")
185+
return html_path
186+
pdf_path = output_dir / f"digest-{today}.pdf"
187+
HTML(string=html).write_pdf(str(pdf_path))
188+
return pdf_path
189+
190+
191+
def email_report(artefact: Path, *,
192+
subject: str,
193+
sender: str, recipient: str,
194+
smtp_host: str, smtp_port: int = 587,
195+
smtp_user: Optional[str] = None,
196+
smtp_pass: Optional[str] = None) -> None:
197+
"""Send ``artefact`` as an attachment via STARTTLS-SMTP."""
198+
message = EmailMessage()
199+
message["Subject"] = subject
200+
message["From"] = sender
201+
message["To"] = recipient
202+
message.set_content(
203+
f"Daily Slack digest attached.\n\nGenerated: {datetime.now().isoformat()}",
204+
)
205+
mime_type = (
206+
"application/pdf" if artefact.suffix == ".pdf"
207+
else "text/html"
208+
)
209+
maintype, subtype = mime_type.split("/", 1)
210+
message.add_attachment(
211+
artefact.read_bytes(), maintype=maintype, subtype=subtype,
212+
filename=artefact.name,
213+
)
214+
context = ssl.create_default_context()
215+
with smtplib.SMTP(smtp_host, smtp_port, timeout=30.0) as server:
216+
server.starttls(context=context)
217+
if smtp_user and smtp_pass:
218+
server.login(smtp_user, smtp_pass)
219+
server.send_message(message)
220+
221+
222+
# --- AutoControl wiring --------------------------------------------
223+
224+
def run_pipeline_once() -> int:
225+
"""Execute one pass of the pipeline; return 0 on success, 1 on error."""
226+
print("Slack daily digest pipeline starting…")
227+
channel = os.environ.get("SLACK_CHANNEL_ID", "C0000000000")
228+
messages = fetch_slack_messages(
229+
channel,
230+
since_hours=24,
231+
token=os.environ.get("SLACK_BOT_TOKEN"),
232+
)
233+
print(f" pulled {len(messages)} messages from {channel}")
234+
235+
summary = summarise(messages)
236+
print(f" summary ready ({len(summary)} chars)")
237+
238+
output_dir = Path("./slack_digests")
239+
artefact = render_report(summary, messages, output_dir)
240+
print(f" artefact: {artefact}")
241+
242+
smtp_host = os.environ.get("SMTP_HOST")
243+
sender = os.environ.get("SMTP_FROM")
244+
recipient = os.environ.get("SMTP_TO")
245+
if not (smtp_host and sender and recipient):
246+
print(" (SMTP_* unset — skipping email step; artefact saved locally)")
247+
return 0
248+
try:
249+
email_report(
250+
artefact,
251+
subject=f"Slack daily digest — {datetime.now().date()}",
252+
sender=sender, recipient=recipient,
253+
smtp_host=smtp_host,
254+
smtp_port=int(os.environ.get("SMTP_PORT", "587")),
255+
smtp_user=os.environ.get("SMTP_USER"),
256+
smtp_pass=os.environ.get("SMTP_PASS"),
257+
)
258+
except (OSError, smtplib.SMTPException) as error:
259+
print(f" email failed: {error}")
260+
return 1
261+
print(f" emailed {artefact.name} to {recipient}")
262+
return 0
263+
264+
265+
def schedule_daily(hour: int = 18, minute: int = 0) -> None:
266+
"""Register the pipeline with the AutoControl scheduler.
267+
268+
Uses a cron expression so the firing time survives process restarts.
269+
"""
270+
bridge_script = Path(__file__).with_name("18_slack_pipeline_bridge.json")
271+
bridge_script.write_text(
272+
json.dumps([
273+
["AC_shell_command", {
274+
"command": f"{sys.executable} {Path(__file__).resolve()} --run",
275+
}],
276+
]),
277+
encoding="utf-8",
278+
)
279+
ac.default_scheduler.add_cron_job(
280+
script_path=str(bridge_script),
281+
cron_expression=f"{minute} {hour} * * *",
282+
job_id="slack-daily-digest",
283+
)
284+
ac.default_scheduler.start()
285+
print(f"scheduled at {hour:02d}:{minute:02d} UTC daily — Ctrl-C to stop.")
286+
287+
288+
def main() -> int:
289+
if "--run" in sys.argv[1:]:
290+
return run_pipeline_once()
291+
if "--schedule" in sys.argv[1:]:
292+
schedule_daily()
293+
import time
294+
try:
295+
while True:
296+
time.sleep(60)
297+
except KeyboardInterrupt:
298+
ac.default_scheduler.stop()
299+
return 0
300+
# Default: run once and exit.
301+
return run_pipeline_once()
302+
303+
304+
if __name__ == "__main__":
305+
sys.exit(main())

examples/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,12 @@ glue between AutoControl's public API and your code.
3636
| [`16_secrets.py`](16_secrets.py) | Store and read credentials from the Fernet-encrypted secret vault. |
3737
| [`17_plugin_loading.py`](17_plugin_loading.py) | Load extra `AC_*` commands from an external plugin file. |
3838

39+
## End-to-end pipeline
40+
41+
| Script | What it shows |
42+
| --- | --- |
43+
| [`18_slack_daily_report.py`](18_slack_daily_report.py) | Full daily workflow: pull Slack messages → Anthropic summary → HTML/PDF report → SMTP email, on a cron schedule. Every external service degrades to a stub so the script always runs to completion. |
44+
3945
## Running
4046

4147
Each script is standalone and uses only the package facade

0 commit comments

Comments
 (0)