Published on 2025-08-07T06:18:08Z

NetcraftSurveyAgent

NetcraftSurveyAgent is a specialized web crawler from Netcraft, an internet security and infrastructure analysis company. Its purpose is to scan websites to collect technical data about their web server technologies, SSL configurations, and hosting infrastructure. This data is used for Netcraft's market intelligence reports and security analysis services. The bot is designed to be non-invasive and respects server resources.

What is NetcraftSurveyAgent?

NetcraftSurveyAgent is the official web crawler for Netcraft, a company known for its internet security and infrastructure analysis. The bot systematically visits websites to collect technical information, such as web server software, SSL certificate details, and hosting infrastructure. It identifies itself with the user-agent string Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected]). The bot is designed to be non-invasive, with a built-in delay between requests, and it does not process dynamic content like JavaScript, as its focus is on server-side data.

Why is NetcraftSurveyAgent crawling my site?

NetcraftSurveyAgent is crawling your site to gather technical data about your web infrastructure for its internet surveys. It is not interested in your page content but rather in the technical details revealed by your server. It scans sites across various sectors, with a particular focus on government and financial websites for security auditing. Most sites will see only occasional visits as part of Netcraft's regular survey updates.

What is the purpose of NetcraftSurveyAgent?

The purpose of NetcraftSurveyAgent is to support Netcraft's commercial services, which include internet security monitoring, anti-phishing protection, and market intelligence. The data it collects contributes to reports on internet trends, such as shifts in web server market share and TLS implementation. Financial institutions and cybersecurity firms use these insights for trend analysis and risk assessment. For website owners, the bot's activity has minimal impact while contributing to broader internet security and technology tracking efforts.

How do I block NetcraftSurveyAgent?

To prevent NetcraftSurveyAgent from accessing your website, you can add a disallow rule to your robots.txt file. This is the standard method for managing crawler access.

Add the following lines to your robots.txt file to block this bot:

User-agent: NetcraftSurveyAgent
Disallow: /

How to verify the authenticity of the user-agent operated by Netcraft?

Reverse IP lookup technique

To verify user-agent authenticity, you can use host linux command two times with the IP address of the requester.
  1. > host IPAddressOfRequest
    This command returns the reverse lookup hostname (e.g., 4.4.8.8.in-addr.arpa.). 
  2. > host ReverseDNSFromTheOutputOfFirstRequest
If the output matches the original IP address and the domain is associated with a trusted operator (e.g., Netcraft), the user-agent can be considered legitimate.

IP list lookup technique

Some operators provide a public list of IP addresses used by their crawlers. This list can be cross-referenced to verify a user-agent's authenticity. However, both operators and website owners may find it challenging to maintain an up-to-date list, so use this method with caution and in conjunction with other verification techniques.