Published on 2025-08-07T06:18:08Z

NewsBlur bot

The NewsBlur bot is the content fetcher for NewsBlur, a popular RSS feed reader and news aggregation service. The bot visits websites to retrieve content from the RSS/Atom feeds that NewsBlur users have subscribed to. Its presence on your site is a direct indicator of user interest and helps expand your content's reach to a dedicated audience of feed readers. A unique feature is that its user-agent string includes the number of users subscribed to your feed.

What is the NewsBlur bot?

The NewsBlur bot is the official web crawler for the RSS feed reader service NewsBlur. It functions as a specialized fetcher for RSS/Atom feeds and webpage content. It operates by regularly checking subscribed feeds for new content, which it then processes and displays to users. A distinctive characteristic is its user-agent string, which includes the number of users subscribed to a particular feed, for example: NewsBlur Feed Fetcher - 7 subscribers - .... This provides transparent feedback to website owners about their readership on the platform.

Why is the NewsBlur bot crawling my site?

The NewsBlur bot is crawling your site because at least one NewsBlur user has subscribed to your site's RSS/Atom feed. The crawler is fetching new content to keep that user's feed updated. The frequency of visits is determined by how often you update your content and how many users subscribe to your feed. This is an authorized and expected activity for any website that publishes content in an RSS format designed for syndication.

What is the purpose of the NewsBlur bot?

The purpose of the NewsBlur bot is to power the NewsBlur content aggregation platform. It helps users efficiently consume content from multiple sources in one place. For website owners, NewsBlur provides value by expanding your content's reach to a dedicated audience. The subscriber count included in the user-agent string also offers a unique and valuable insight into how many users are following your content through the service.

How do I block the NewsBlur bot?

If you wish to prevent NewsBlur from fetching your content feeds, you can add a rule to your robots.txt file. Note that this will stop NewsBlur users from receiving updates from your site. There are several user-agents for the service, but the main one for feeds is NewsBlur Feed Fetcher.

To block this fetcher, add the following lines to your robots.txt file:

User-agent: NewsBlur Feed Fetcher
Disallow: /

How to verify the authenticity of the user-agent operated by NewsBlur?

Reverse IP lookup technique

To verify user-agent authenticity, you can use host linux command two times with the IP address of the requester.
  1. > host IPAddressOfRequest
    This command returns the reverse lookup hostname (e.g., 4.4.8.8.in-addr.arpa.). 
  2. > host ReverseDNSFromTheOutputOfFirstRequest
If the output matches the original IP address and the domain is associated with a trusted operator (e.g., NewsBlur), the user-agent can be considered legitimate.

IP list lookup technique

Some operators provide a public list of IP addresses used by their crawlers. This list can be cross-referenced to verify a user-agent's authenticity. However, both operators and website owners may find it challenging to maintain an up-to-date list, so use this method with caution and in conjunction with other verification techniques.