Microsoft Clarity has announced that the tool’s Bot Analytics dashboard now displays bot requests that violate a website’s URL rules.
Clarity will determine and show these requests as a proportion of overall bot engagement during a specified period and enhance the current AI Visibility tools on the dashboard, which recently started presenting the foundational queries supporting AI references.
What Can Be Seen in the Violations Perspective
When a bot sends a request to a website linked to Clarity, the tool verifies the request with the site’s robots.txt rules to see if the path is prohibited.
Bot requests that are not permitted are calculated and shown as a percentage of overall bot activity within a specific time period.
Site owners can use clarity to filter bot requests based on various criteria and analyze patterns in crawler behavior.
This involves comparing compliant crawlers with those displaying violations in a side-by-side view.
How to activate it
The feature needs to be manually activated by a project admin in the AI Visibility section of Project Settings for sites that use a supported CDN.
Supported content delivery networks (CDNs) include Fastly, Amazon CloudFront, Cloudflare, Azure Front Door, and Akamai. WordPress websites utilizing the most recent Microsoft Clarity plugin are also compatible.
Why This Is Important
Being able to observe this activity is important due to worries about AI crawlers consuming server resources and distorting analytics.
Clarity is free to use and can help monitor if crawlers are following the rules by indicating when the requests were made, without providing the reasons behind them.
Robots.txt is a guide that doesn’t block anything, so the data here focuses on requests that were allowed to access paths restricted by a site’s robots.txt.
Manually analyzing server logs for bot requests and testing URLs against robots.txt to find disallowed requests is not efficient. Clarity now automatically tracks the number of requests from crawlers that violate a site’s rules, recognizing the limitations of manual methods.
Looking to the future
Websites can now use more precise, automated methods to evaluate compliance with robots.txt rules.
The main question is if simplifying the measurement of this behavior will impact how crawlers act, or if it simply assists website owners in maintaining a more transparent record of activities.
