Show HN: SoMatic – Vision-based OS automation framework for AI agents
SoMatic is a command-line interface designed for desktop UI automation using AI agents. It utilizes a YOLO model to identify and number interactive elements in screenshots, allowing agents to perform actions based on structured coordinates. The framework supports various commands for automating tasks across different applications and platforms.
- ▪SoMatic runs a local YOLO model to detect interactive elements in screenshots.
- ▪Commands return JSON, enabling structured interactions with desktop applications.
- ▪The framework is compatible with Windows, macOS, and Linux, each requiring specific permissions.
Opening excerpt (first ~120 words) tap to expand
SoMatic SoMatic is an agent-first CLI for native desktop UI automation. It runs a local YOLO model to detect and number every interactive element in a screenshot, giving the agent a structured coordinate map it can ground actions against. Elements can be targeted by mark ID, by nearest-mark offset, or by direct pixel coordinate — no guessing. Every command returns JSON. The public binary is somatic. Demo Watch the higher-quality MP4 demo: docs/assets/somatic-demo.mp4. SoMatic turns desktop screenshots into numbered action targets that agents can use across native apps, browsers, PDFs, and web tools: PDF workflow Chess setup Browser + terminal Install npm install -g @somatic-cli/cli The npm package launches the Python core. Python 3.10+ must be available on PATH.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at GitHub.