An unseen, non-human hand moving the cursor across your computer screen and typing without using the keyboard in fiction is usually a sign of malicious AI hijacking something (or a friendly ghost helping you solve mysteries like the TV show Ghost Writer). Thanks to Anthropic’s new computer use feature for its AI assistant Claude, there’s a much more benevolent explanation now.
Fueled by an upgraded version of the Claude 3.5 Sonnet model, this AI – dubbed ‘computer use’ – lets you interact with your computer much like you would. It takes the AI assistant concept a step beyond text and a voice, with virtual hands typing, clicking, and otherwise manipulating your computer.
Anthropic bills computer use as a way for Claude to handle tedious tasks. It can help you fill out a form, search and organize information on your hard drive, and move information around. While OpenAI, Microsoft, and other developers have demonstrated similar ideas, Anthropic is the first to have a public feature, though it’s still in beta.
“With computer use, we’re trying something fundamentally new,” Anthropic explained in a blog post. Instead of making specific tools to help Claude complete individual tasks, we’re teaching it general computer skills—allowing it to use a wide range of standard tools and software programs designed for people.”
The computer use feature is due to Claude 3.5 Sonnet’s improved performance, particularly with digital tools and coding software. Though somewhat overshadowed by the spectacle of the computer use feature, Anthropic also debuted a new model called Claude 3.5 Haiku, a more advanced version of the lower-cost Anthropic model, though once capable of matching Anthropic’s previous highest performing model, Claude 3 Opus, while still being much cheaper.
Watch On
Invisible AI assistance
You can’t just give an order and walk away, either. Claude’s control of your computer has some technical troubles as well as deliberate constraints. On the technical side, Anthropic admitted Claude struggles with scrolling and zooming around a screen. That’s because the AI interprets what’s on your screen as a collection of screenshots, and then it tries to piece them together like a movie reel. Anything that happens too quickly or that changes perspective on the screen can flummox it. Still, Claude can do quite a lot by manipulating your computer, as seen above.
Unrestrained automation has obvious perils even when working perfectly, as so many sci-fi movies and books have explored. Claude isn’t Skynet, but Anthropic has placed restraints on the AI for more prosaic reasons. For instance, there are guardrails stopping Claude from interacting with social media or any government websites. Registering domain names or posting content is not allowed without human control.
“Because computer use may provide a new vector for more familiar threats such as spam, misinformation, or fraud, we’re taking a proactive approach to promote its safe deployment. We’ve developed new classifiers that can identify when computer use is being used and whether harm is occurring,” Anthropic wrote. “Learning from the initial deployments of this technology, which is still in its earliest stages, will help us better understand both the potential and the implications of increasingly capable AI systems.”
