Skip to content

Add Linux (X11 + Wayland) support to coven-desktop-use #1

@matedev01

Description

@matedev01

Summary

coven-desktop-use currently supports macOS only (via Peekaboo) and returns
"unsupported" on Linux/Windows. Requesting a Linux backend (X11 + Wayland) so
the OpenClaw desktop_use tool works for users running Ubuntu / Linux as
their primary OS.

Motivation

Several OpenCoven users run Linux as their daily driver (embedded engineering,
headless agents, dev VMs, CI). Without a Linux backend the desktop_use tool
has to be disabled in plugin config, which removes a significant capability
from the agent on those machines.

Scope

Lean v1 that mirrors the existing JSON envelope and --confirm policy, with
per-session backends instead of a single bundled tool:

  • X11: scrot / maim (capture), xdotool (input), wmctrl (focus)
  • Wayland: grim (capture), wtype / ydotool (input), swaymsg (focus on Sway)

doctor returns a tool inventory and the exact apt install line for any
missing pieces. macOS path is unchanged.

Out of scope (deferred)

  • AT-SPI element-tree annotation — Linux inspect returns a screenshot but
    no B1/T2 element ids in v1. Callers use --coords x,y instead.
  • Active-window capture on vanilla Wayland (grim has no concept of focused
    window).
  • Real scroll-wheel events on Wayland — degrades to Page_Up/Page_Down
    via wtype, marked degraded in the response.
  • Window focus on GNOME Mutter / KDE KWin — no public CLI exists.

Compatibility

  • macOS path unchanged.
  • TS plugin schema unchanged; new fields are additive.
  • Typed-text redaction extended to cover xdotool/wtype's -- separator.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions