Google has added a “Computer Use” function to the Mac version of Gemini. It can click, type, navigate menus, fill out forms, and complete multi-step tasks across local files, desktop applications, and web pages on your Mac.
There are two thresholds: people in the United States, speaking English; and subscribing to one of Google AI Plus, Pro or Ultra. At present, the volume is gradually increased, and it is normal that it has not been brushed.
The best thing about the design is that background window. Gemini does not steal your screen or cursor when working. It runs in a separate background window. There is a real-time preview in the chat interface, so you can watch it while doing your own thing.
Control is given relatively finely. You can choose which applications it is allowed to access, and it can be paused at any time. It will stop and wait for your approval before critical operations.
Examples Google gives include filling out online forms, making presentations, doing creative work, and putting products up for sale.
This feature doesn’t come too early. ChatGPT has long had computer control tools, Anthropic has Duet, and Meta’s Muse has long been ranked second on the App Store’s free list. Google just this week renamed its own robot Tasks, which can make calls for people.
The timing is subtle. On the same day, it was reported that Apple is recruiting people to be its own general agent, and in the upcoming macOS 27.3 beta version, the steps to enable ChatGPT computer control have been changed from 4 steps to 14 steps.
The road for third-party smartphones on Mac is being narrowed by Apple. Gemini squeezed in at this time, using the same set of system permissions. It will be worth watching whether those permission warnings will also fall on it.
Ordinary users can try low-risk things first, such as organizing download folders and filling out a form that does not involve payment. To allow a model to access both the browser and local files, it is best to go through the permissions list yourself.