Published on 2025-08-07T06:18:08Z

Feedfetcher-Google

Feedfetcher-Google is a specialized web crawler from Google that retrieves RSS and Atom feeds on behalf of users. It is not an autonomous web crawler but a user-triggered fetcher; it only visits a feed URL when a user has explicitly subscribed to it through a Google service like Google News. Its purpose is to efficiently deliver updated feed content to users while reducing bandwidth consumption for publishers.

What is Feedfetcher-Google?

Feedfetcher-Google is an official crawler from Google designed specifically to retrieve RSS and Atom feeds from websites. It is classified as a user-triggered fetcher, meaning it does not autonomously discover content. Instead, it activates only when users subscribe to a feed through a Google service, such as Google News, or when publishers use technologies like PubSubHubbub. The bot identifies itself in server logs with the user-agent string Feedfetcher-Google. It operates on a distributed system to improve performance and efficiently deliver feed updates.

Why is Feedfetcher-Google crawling my site?

Feedfetcher-Google is visiting your site because one or more users have subscribed to your RSS or Atom feed through a Google service. The visit is always initiated by a user's action and is not part of an automated web crawl. The frequency of its visits depends on how often your feed is updated and the number of subscribers you have through Google's services. It generally should not retrieve feeds more than once per hour on average. Its presence is a sign of direct user interest in your syndicated content.

What is the purpose of Feedfetcher-Google?

The primary purpose of Feedfetcher-Google is to act as an intermediary that delivers requested feed content to users of Google services. It retrieves, stores, and refreshes RSS or Atom feeds on behalf of users. Unlike general web crawlers, it typically does not index the content it collects for Google Search (with the exception of podcasts). By fetching a feed once for multiple subscribers, Feedfetcher helps conserve bandwidth for publishers while still providing timely updates to users. For publishers, its activity indicates an active and engaged audience within the Google ecosystem.

How do I block Feedfetcher-Google?

If you wish to prevent Google's feed fetching service from accessing your RSS or Atom feeds, you can add a rule to your robots.txt file. Note that this will prevent users from subscribing to and receiving updates from your feeds through Google's services.

To block Feedfetcher-Google, add the following lines to your robots.txt file:

User-agent: Feedfetcher-Google
Disallow: /

How to verify the authenticity of the user-agent operated by Google?

Reverse IP lookup technique

To verify user-agent authenticity, you can use host linux command two times with the IP address of the requester.

```
> host IPAddressOfRequest
```
This command returns the reverse lookup hostname (e.g., 4.4.8.8.in-addr.arpa.).

> host ReverseDNSFromTheOutputOfFirstRequest

If the output matches the original IP address and the domain is associated with a trusted operator (e.g., Google), the user-agent can be considered legitimate.

IP list lookup technique

Some operators provide a public list of IP addresses used by their crawlers. This list can be cross-referenced to verify a user-agent's authenticity. However, both operators and website owners may find it challenging to maintain an up-to-date list, so use this method with caution and in conjunction with other verification techniques.