linkdexbot

linkdexbot is the official web crawler for the enterprise SEO platform Authoritas (which acquired Linkdex). It is a technical SEO crawler that systematically browses websites to collect data for SEO auditing and analysis. A key characteristic is that it operates only under the explicit direction of an Authoritas client; it does not crawl the web autonomously. The bot identifies itself in server logs with user-agent strings like Mozilla/5.0 (compatible; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/).

The presence of linkdexbot in your server logs indicates that an Authoritas client, such as an SEO professional or marketing agency, is actively analyzing your website. The bot is examining your site's structure, content, and technical elements to provide SEO insights to that user. The frequency of visits depends on the client's settings and the scope of their analysis. While the crawl is initiated by a third party, it is generally considered authorized as part of legitimate SEO research.

The purpose of linkdexbot is to serve the Authoritas SEO platform by collecting the data that helps users understand and improve a website's search engine optimization. It supports backlink analysis by mapping link graphs and conducts technical audits to detect issues like broken links or non-canonical URLs. For website owners, the indirect value of this crawling comes from the insights that SEO professionals gain from the data, which can lead to recommendations that improve your site's performance in search results.

To prevent linkdexbot from analyzing your site, you can add a disallow rule to your robots.txt file. This is the standard method for managing access for legitimate crawlers.

To block linkdexbot, add the following lines to your robots.txt file:

User-agent: linkdexbot
Disallow: /