Published on 2025-08-07T06:18:08Z

DomainStatsBot

DomainStatsBot is a web crawler operated by Domainstats International AB. It systematically scans websites to collect public, domain-related data and statistics. This information powers DomainStats' analytical services, which are used for purposes such as competitive analysis and domain intelligence. The bot focuses on basic site structure and content and does not process interactive elements like JavaScript.

What is DomainStatsBot?

DomainStatsBot is the official web crawler for Domainstats International AB. It is an automated bot that browses websites to collect data and statistics related to internet domains. The bot identifies itself in server logs with the user-agent string DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot), which transparently provides a link to its documentation. It is a general web crawler designed to perform HTTP requests to gather information on domain properties and usage patterns, but it does not process interactive elements like JavaScript or cookies.

Why is DomainStatsBot crawling my site?

DomainStatsBot is visiting your website to gather public, domain-specific information for its analytical services. The data it collects may include your domain's characteristics, content structure, and other technical metrics that contribute to domain statistics and analysis. The frequency of its visits typically depends on your site's size, popularity, and update schedule. The crawling is part of a legitimate business activity, and the bot's clear identification indicates it is not attempting to conceal its purpose.

What is the purpose of DomainStatsBot?

The main purpose of DomainStatsBot is to serve as the data collection mechanism for DomainStats' domain intelligence and analysis platform. It gathers information to build comprehensive domain profiles, which may include metrics on domain popularity, content types, and technical configurations. This data likely supports various analytical tools offered by DomainStats. The value of this data flows primarily to the users of DomainStats' services rather than to the owners of the websites being crawled.

How do I block DomainStatsBot?

To prevent DomainStatsBot from accessing your website, you can add a disallow rule for it in your robots.txt file. This is the standard method for managing access for well-behaved web crawlers.

Add the following lines to your robots.txt file to block DomainStatsBot:

User-agent: DomainStatsBot
Disallow: /

How to verify the authenticity of the user-agent operated by Domainstats?

Reverse IP lookup technique

To verify user-agent authenticity, you can use host linux command two times with the IP address of the requester.
  1. > host IPAddressOfRequest
    This command returns the reverse lookup hostname (e.g., 4.4.8.8.in-addr.arpa.). 
  2. > host ReverseDNSFromTheOutputOfFirstRequest
If the output matches the original IP address and the domain is associated with a trusted operator (e.g., Domainstats), the user-agent can be considered legitimate.

IP list lookup technique

Some operators provide a public list of IP addresses used by their crawlers. This list can be cross-referenced to verify a user-agent's authenticity. However, both operators and website owners may find it challenging to maintain an up-to-date list, so use this method with caution and in conjunction with other verification techniques.