I built the Playwright for desktop apps. 80% token savings
The article introduces agent-desktop, a Rust-based CLI tool for automating desktop applications using accessibility APIs without relying on screenshots or browser environments. It enables AI agents to interact with any app on the system through structured commands and optimized workflows, significantly reducing token usage. The tool supports multiple programming languages via a C-ABI FFI interface and is available for installation via npm or from source.
- ▪agent-desktop is a native Rust CLI that provides structured access to desktop applications through OS accessibility trees.
- ▪It supports progressive skeleton traversal, achieving 78–96% token reduction in dense applications like Slack and VS Code.
- ▪The tool offers language bindings for Python, Swift, Go, Ruby, Node, and C via a prebuilt C-ABI cdylib library.
- ▪Structured JSON output and deterministic element references (@e1, @e2) enable reliable AI-driven automation workflows.
- ▪agent-desktop requires Accessibility permissions on macOS and is compatible with apps such as Finder, Safari, System Settings, Xcode, and Slack.
Opening excerpt (first ~120 words) tap to expand
AGENT DESKTOP OBSERVE. DECIDE. ACT. agent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required. Architecture Key Features Native Rust CLI: Fast, single binary, no runtime dependencies C-ABI cdylib (libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call 53 commands: Observation, interaction, keyboard, mouse, notifications, clipboard, window management Progressive skeleton traversal: 78–96% token reduction on dense apps via shallow overview + targeted drill-down Snapshot & refs: AI-optimized workflow using deterministic element references (@e1, @e2) AX-first interactions:…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Hacker News: Front Page.
Discussion
0 commentsMore from Hacker News: Front Page
