Skip to content

Commit caf6514

Browse files
committed
Add 23 new features across locator, observability, IDE, platform layers
Locator + selector intelligence - Self-healing locator: image template → VLM fallback with audit log - Anchor-based locator: find element B by spatial relation to anchor A - OCR with structured output: detect rows / tables / form-field pairs - Smart waits: wait_until_screen_stable, _pixel_changes, _region_idle - A/B locator framework: race N strategies, recommend the historical best Operations + observability - Cost telemetry: per-call LLM token + USD log with day/model/provider rollup - Trace replay UI: scrubbable timeline over the time-travel recordings - Failure → ticket automation: Jira / Linear / GitHub fan-out on run failures - Container CI: GH Actions + GitLab templates, XFCE+VNC Dockerfile variant - Cross-host DAG orchestrator: parallel execution with skip-on-failure cascade - Multi-viewer presence: roster + controller/observer roles for remote desktop Agent + integrations - Computer-use high-level API: wraps ComputerUseAgentBackend + AgentLoop - WebRunner executor + MCP integration: AC_web_open/quit/screenshot helpers - Chat-ops bot: transport-agnostic CommandRouter + Slack polling adapter Platform coverage - Wayland CLI backend: wtype + ydotool + grim with auto-detect + X11 fallback - Wayland libei native backend: ctypes binding, opt-in via env override - macOS Accessibility: tree dump + polling event recorder Developer experience - autocontrol-lsp: didOpen/didChange/didClose, diagnostics, signature help - .pyi stub generator: introspects Executor.event_dict for IDE autocomplete - VS Code extension: LSP client + Run/Screenshot/Preview REST commands - Browser extension recorder: MV3 capture → AC_web_run JSON export - pytest plugin + Gherkin BDD: fixtures, @AutoControl marker, step library - Visual flow editor: node-based view round-trips to JSON action format Surfaces wired uniformly per CLAUDE.md feature-delivery rules: - headless API in utils/ with zero PySide6 imports - executor commands (AC_*) registered in action_executor.py - MCP tools (ac_*) registered in mcp_server/tools/_factories.py - GUI tab for interactive features, all i18n'd across en/zh-TW/zh-CN/ja - facade re-exports in je_auto_control/__init__.py - headless tests; full suite stays green with no regressions
1 parent 8a0b13e commit caf6514

127 files changed

Lines changed: 16948 additions & 104 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/docker.yml

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
name: AutoControl Docker CI
2+
3+
on:
4+
push:
5+
branches: [ "dev", "main" ]
6+
paths:
7+
- "docker/**"
8+
- "je_auto_control/**"
9+
- "pyproject.toml"
10+
- ".github/workflows/docker.yml"
11+
pull_request:
12+
branches: [ "dev", "main" ]
13+
paths:
14+
- "docker/**"
15+
- "je_auto_control/**"
16+
- "pyproject.toml"
17+
- ".github/workflows/docker.yml"
18+
19+
permissions:
20+
contents: read
21+
22+
jobs:
23+
build-image:
24+
name: Build AutoControl container
25+
runs-on: ubuntu-22.04
26+
27+
steps:
28+
- uses: actions/checkout@v4
29+
30+
- name: Set up Docker Buildx
31+
uses: docker/setup-buildx-action@v3
32+
33+
- name: Build image (no push)
34+
uses: docker/build-push-action@v5
35+
with:
36+
context: .
37+
file: docker/Dockerfile
38+
tags: autocontrol:ci
39+
load: true
40+
cache-from: type=gha
41+
cache-to: type=gha,mode=max
42+
43+
- name: Image size
44+
run: docker image inspect autocontrol:ci --format='size={{.Size}} bytes'
45+
46+
headless-tests:
47+
name: Headless pytest inside the image
48+
needs: build-image
49+
runs-on: ubuntu-22.04
50+
51+
steps:
52+
- uses: actions/checkout@v4
53+
54+
- name: Set up Docker Buildx
55+
uses: docker/setup-buildx-action@v3
56+
57+
- name: Rebuild image (cached)
58+
uses: docker/build-push-action@v5
59+
with:
60+
context: .
61+
file: docker/Dockerfile
62+
tags: autocontrol:ci
63+
load: true
64+
cache-from: type=gha
65+
66+
# Mount the repo so pytest can read tests + write the artifact.
67+
- name: Run headless tests under Xvfb
68+
run: |
69+
docker run --rm \
70+
--user root \
71+
-v "$PWD:/work" -w /work \
72+
--entrypoint /bin/sh \
73+
autocontrol:ci -c "
74+
pip install --no-cache-dir -r dev_requirements.txt &&
75+
xvfb-run -a -s '-screen 0 1280x800x24' \
76+
python -m pytest test/unit_test/headless -q --tb=short
77+
"
78+
79+
- name: Smoke test the entrypoint (rest mode)
80+
run: |
81+
docker run --rm -d --name ac-rest -p 9939:9939 \
82+
-e AC_TOKEN=ci-token autocontrol:ci rest
83+
for attempt in 1 2 3 4 5 6 7 8 9 10; do
84+
if curl -fsS -H "Authorization: Bearer ci-token" \
85+
http://127.0.0.1:9939/health; then
86+
echo "REST API is up"
87+
break
88+
fi
89+
sleep 2
90+
done
91+
docker logs ac-rest || true
92+
docker stop ac-rest
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
"""Build LSP ``Diagnostic`` lists for an AutoControl action JSON file.
2+
3+
Two layers of checking:
4+
5+
1. **JSON parse**: a parse failure surfaces as a diagnostic at the
6+
reported error line, with severity ``Error``;
7+
2. **Schema**: top-level must be a list; each entry must be a 1- or
8+
2-element list ``[name]`` / ``[name, params_dict]`` where ``name``
9+
is a registered ``AC_*`` command and ``params`` is an object.
10+
11+
Diagnostics are returned in the LSP wire shape — the server hands
12+
them straight to ``textDocument/publishDiagnostics``.
13+
"""
14+
from __future__ import annotations
15+
16+
import json
17+
from typing import Any, Dict, List, Optional
18+
19+
from autocontrol_lsp.server.commands import known_action_names
20+
21+
22+
_SEVERITY_ERROR = 1
23+
_SEVERITY_WARNING = 2
24+
25+
26+
def diagnostics_for(text: str) -> List[Dict[str, Any]]:
27+
"""Return every problem found in ``text`` as an LSP Diagnostic dict."""
28+
try:
29+
data = json.loads(text or "")
30+
except json.JSONDecodeError as error:
31+
return [_parse_error_diagnostic(error)]
32+
return _schema_diagnostics(data)
33+
34+
35+
def _parse_error_diagnostic(error: json.JSONDecodeError) -> Dict[str, Any]:
36+
line = max(0, int(error.lineno) - 1)
37+
column = max(0, int(error.colno) - 1)
38+
return {
39+
"range": {
40+
"start": {"line": line, "character": column},
41+
"end": {"line": line, "character": column + 1},
42+
},
43+
"severity": _SEVERITY_ERROR,
44+
"source": "autocontrol-lsp",
45+
"message": f"invalid JSON: {error.msg}",
46+
}
47+
48+
49+
def _schema_diagnostics(data: Any) -> List[Dict[str, Any]]:
50+
out: List[Dict[str, Any]] = []
51+
if not isinstance(data, list):
52+
out.append(_root_must_be_list_diagnostic())
53+
return out
54+
known = set(known_action_names())
55+
for index, entry in enumerate(data):
56+
problem = _check_entry(entry, known)
57+
if problem is None:
58+
continue
59+
out.append(_diagnostic_for_entry(index, problem))
60+
return out
61+
62+
63+
def _check_entry(entry: Any, known: set) -> Optional[str]:
64+
if not isinstance(entry, list):
65+
return "action must be a list of [name] or [name, params]"
66+
if not entry:
67+
return "action list cannot be empty"
68+
name = entry[0]
69+
if not isinstance(name, str):
70+
return f"action name must be a string, got {type(name).__name__}"
71+
if not name.startswith("AC_"):
72+
return f"action name {name!r} must start with AC_"
73+
if name not in known:
74+
return f"unknown AC_ command: {name!r}"
75+
if len(entry) > 2:
76+
return "action accepts at most [name, params]"
77+
if len(entry) == 2 and not isinstance(entry[1], dict):
78+
return "params must be an object"
79+
return None
80+
81+
82+
def _diagnostic_for_entry(index: int, message: str) -> Dict[str, Any]:
83+
return {
84+
"range": {
85+
"start": {"line": 0, "character": 0},
86+
"end": {"line": 0, "character": 1},
87+
},
88+
"severity": _SEVERITY_WARNING,
89+
"source": "autocontrol-lsp",
90+
"message": f"action[{index}]: {message}",
91+
}
92+
93+
94+
def _root_must_be_list_diagnostic() -> Dict[str, Any]:
95+
return {
96+
"range": {
97+
"start": {"line": 0, "character": 0},
98+
"end": {"line": 0, "character": 1},
99+
},
100+
"severity": _SEVERITY_ERROR,
101+
"source": "autocontrol-lsp",
102+
"message": "action file must be a JSON list of [name, params] entries",
103+
}
104+
105+
106+
__all__ = ["diagnostics_for"]
Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
"""In-memory document store for the LSP server.
2+
3+
LSP clients send ``textDocument/didOpen`` with the full file contents
4+
and ``didChange`` with either full or incremental updates. The server
5+
needs the *current* text whenever hover / completion / diagnostics is
6+
requested — that's what this module owns. Pure stdlib, thread-safe.
7+
"""
8+
from __future__ import annotations
9+
10+
import threading
11+
from dataclasses import dataclass
12+
from typing import Dict, List, Optional, Sequence
13+
14+
15+
@dataclass(frozen=True)
16+
class Position:
17+
"""Zero-based ``(line, character)`` LSP position."""
18+
19+
line: int
20+
character: int
21+
22+
23+
@dataclass(frozen=True)
24+
class TextDocument:
25+
"""Versioned text snapshot keyed by URI."""
26+
27+
uri: str
28+
text: str
29+
version: int = 0
30+
31+
def lines(self) -> List[str]:
32+
return self.text.splitlines()
33+
34+
def word_at(self, position: Position) -> str:
35+
"""Return the identifier-like word under ``position`` (LSP style)."""
36+
rows = self.lines()
37+
if position.line < 0 or position.line >= len(rows):
38+
return ""
39+
line = rows[position.line]
40+
index = position.character
41+
if index < 0 or index > len(line):
42+
return ""
43+
return _word_around(line, min(index, len(line)))
44+
45+
46+
class DocumentStore:
47+
"""Thread-safe ``uri → TextDocument`` map for the LSP loop."""
48+
49+
def __init__(self) -> None:
50+
self._docs: Dict[str, TextDocument] = {}
51+
self._lock = threading.RLock()
52+
53+
def open(self, uri: str, text: str, version: int = 0) -> TextDocument:
54+
doc = TextDocument(uri=str(uri), text=str(text), version=int(version))
55+
with self._lock:
56+
self._docs[doc.uri] = doc
57+
return doc
58+
59+
def replace(self, uri: str, text: str,
60+
version: Optional[int] = None) -> TextDocument:
61+
with self._lock:
62+
existing = self._docs.get(uri)
63+
next_version = (
64+
int(version) if version is not None
65+
else (existing.version + 1 if existing else 0)
66+
)
67+
return self.open(uri, text, next_version)
68+
69+
def close(self, uri: str) -> bool:
70+
with self._lock:
71+
return self._docs.pop(uri, None) is not None
72+
73+
def get(self, uri: str) -> Optional[TextDocument]:
74+
with self._lock:
75+
return self._docs.get(uri)
76+
77+
def count(self) -> int:
78+
with self._lock:
79+
return len(self._docs)
80+
81+
def apply_change(self, uri: str,
82+
changes: Sequence[Dict],
83+
version: Optional[int] = None,
84+
) -> Optional[TextDocument]:
85+
"""Apply LSP ``contentChanges`` entries to the stored document.
86+
87+
Supports both full-document and range-incremental forms.
88+
Returns the updated document, or None when the URI isn't tracked.
89+
"""
90+
with self._lock:
91+
doc = self._docs.get(uri)
92+
if doc is None:
93+
return None
94+
text = doc.text
95+
for change in changes:
96+
if "range" not in change:
97+
text = str(change.get("text", ""))
98+
continue
99+
text = _apply_range_edit(text, change["range"],
100+
str(change.get("text", "")))
101+
return self.replace(uri, text, version=version)
102+
103+
104+
# --- helpers -------------------------------------------------
105+
106+
_WORD_CHARS = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_")
107+
108+
109+
def _word_around(line: str, index: int) -> str:
110+
if index >= len(line):
111+
index = len(line) - 1
112+
if index < 0:
113+
return ""
114+
if line[index] not in _WORD_CHARS:
115+
# Try the character to the left — common when cursor sits after a name.
116+
if index > 0 and line[index - 1] in _WORD_CHARS:
117+
index -= 1
118+
else:
119+
return ""
120+
start = index
121+
while start > 0 and line[start - 1] in _WORD_CHARS:
122+
start -= 1
123+
end = index
124+
while end + 1 < len(line) and line[end + 1] in _WORD_CHARS:
125+
end += 1
126+
return line[start:end + 1]
127+
128+
129+
def _apply_range_edit(text: str, lsp_range: Dict,
130+
new_text: str) -> str:
131+
start = lsp_range.get("start") or {}
132+
end = lsp_range.get("end") or {}
133+
start_index = _offset_for(text, int(start.get("line", 0)),
134+
int(start.get("character", 0)))
135+
end_index = _offset_for(text, int(end.get("line", 0)),
136+
int(end.get("character", 0)))
137+
if start_index > end_index:
138+
start_index, end_index = end_index, start_index
139+
return text[:start_index] + new_text + text[end_index:]
140+
141+
142+
def _offset_for(text: str, line: int, char: int) -> int:
143+
current_line = 0
144+
for index, ch in enumerate(text):
145+
if current_line == line:
146+
return min(len(text), index + char)
147+
if ch == "\n":
148+
current_line += 1
149+
return len(text)
150+
151+
152+
__all__ = ["DocumentStore", "Position", "TextDocument"]

0 commit comments

Comments
 (0)