GitHub has added computer use capabilities to Copilot, allowing it to directly interact with desktop applications.

Copilot can read on-screen content, click controls, type and edit text, scroll, drag, and automate workflows even in legacy software without APIs or command-line support. The feature is currently available in public preview through the GitHub Copilot app on macOS and Windows, as well as Copilot CLI.