Google has integrated “computer use” into Google Gemini 3.5 Flash, allowing for agent-style control of browsers, apps, and desktop workflows as a standard feature rather than a distinct product. This enhancement enables Gemini to observe and engage with user interfaces, analyze on-screen content, and execute direct actions. A senior scientist from Google DeepMind cautioned that large-scale AI agents may encourage malicious activities.
Developers are now able to create agents capable of performing tasks beyond simply calling APIs. They can automate tasks that are limited to the graphical user interface, such as software testing, form filling, dashboard navigation, or using older applications without API integration. This enhances automation efficiency and broadens the scope of tasks AI agents can effectively handle in real-world scenarios.
An AI agent can interact with software that has a graphical user interface (GUI) without an API by performing tasks such as logging into a dashboard, exporting SEO reports to a spreadsheet, comparing data, and emailing a summary using natural language instructions instead of custom scripts.
What SEO Signifies
SEO tools are expected to become more autonomous soon. AI could go beyond providing data and actually perform various tasks like auditing websites, crawling sites, extracting specific data points, and carrying out optimization workflows.
Site owners may need to consider the possibility of AI agents posing as “visitors,” impacting their understanding of site interactions and engagement signals for optimizing site performance and sales.
AI Agents are going to face attacks.
Google’s positive announcement should be noted, especially the importance of following the “safety best practices” document linked to it to prevent theft and ensure a better user experience.
The document explains:
Using a computer involves specific security and operational dangers because a program working for a user may come across unreliable content on displays or make mistakes while carrying out tasks.
The mention of “untrusted content on screens” could be alluding to the “traps” cautioned against by the senior scientist at Google DeepMind for AI agents.
Google suggests seven best practices for using this new AI agent.
Enforce user confirmation is necessary when the safety response indicates it or when a legacy safety decision requires it. Custom safety instructions can be implemented to define and enforce personalized safety boundaries.
Run your agent in a secure, isolated environment to minimize its possible effects. This could involve using a sandboxed virtual machine, a container like Docker, or a dedicated browser profile with restricted permissions.
Sanitize any text input from users in prompts to reduce the chance of unintended commands or prompt injection. This security step is beneficial but should not be seen as a substitute for a secure operating environment.
Utilize guardrails and content safety APIs to assess user inputs, tool inputs and outputs, and the agent’s responses for suitability, prompt injection, and detection of unauthorized access.
Implement filtering mechanisms to regulate the model’s access and actions, starting with a list of banned websites and enhancing security with a stricter list of permitted sites.
Maintain thorough logs for debugging, auditing, and responding to incidents, including prompts, screenshots, model-suggested actions, safety responses, and all client-executed actions.
Ensure that the GUI environment remains uniform to avoid confusion for the model, by minimizing unexpected pop-ups, notifications, or layout changes. Ideally, begin each new task from a familiar, clean state.
Be cautious of websites that are filled with traps.
As attack surfaces increase, hackers are more likely to try to take advantage of them. This means that as more AI agents appear online, hackers will focus on exploiting them, using websites as a battleground to launch attacks on AI agents.
A Google DeepMind senior scientist stated that bad actors are already laying traps to steal money from people through their AI agents.
A cybersecurity professional in California recently had unauthorized charges on his credit card linked to Anthropic Claude’s AI assistant, possibly due to downloading a Skills.md file containing a trap.
The article provides information about…
He discovered a troublesome add-on linked to Claude, described as a “skill,” resembling a plug-in. This add-on directed Claude to try buying various gift accounts using the digital wallet stored on the computer.
Site owners might require more robust bot management tools and the capacity to detect concealed prompt-injection commands left by hackers on their websites. However, this is not a priority for website owners, thereby exacerbating the issue for users employing AI assistants such as Google’s recent release.
Google DeepMind is facing challenges as AI agents are being tricked into losing money.
Image provided by Shutterstock/blocberry
