Controlling the Browser via the Accessibility Tree: A New Approach for AI Agents

The author of the article introduced a Google Chrome extension that allows AI agents to interact with the browser directly through the accessibility tree. Traditional methods, such as taking screenshots or using headless modes, often face challenges: screenshots consume too many tokens when working with LLMs, and headless browsers do not always maintain authorization sessions. The developed solution provides agents with structured access to interface elements, making browser control more efficient and reliable. The tool allows for controlling an open browser via the command line, opening new possibilities for automating complex web scenarios. This approach significantly simplifies the integration of AI into daily workflows that require website navigation, while preserving user context and reducing data processing costs.