Superfeedr bot

Superfeedr is a feed aggregation service, acquired by Medium in 2016, that simplifies how applications handle RSS, Atom, and JSON feeds. Its bot is a specialized feed crawler that continuously polls feeds, detects changes, and pushes updates to subscribers in real-time. This eliminates the need for individual clients to check feeds themselves. The bot identifies itself with a user-agent string like Superfeedr Bot (feed-id=12345), which includes a feed-id that helps trace requests to specific subscriptions.

The Superfeedr bot is crawling your site because someone has subscribed to one of your feeds through a service that uses Superfeedr's API. The bot targets only your feed URLs to check for new content. The frequency of visits depends on how often you update your content and the number of subscribers you have through these services. This crawling is an authorized and standard activity for any website that offers public feeds for content syndication.

The purpose of Superfeedr is to make the distribution of web content updates more efficient. It acts as a middleware layer that reduces the inefficiencies of traditional feed polling, where thousands of individual readers might check the same feed. For website owners, this provides the indirect benefit of reducing server load, as multiple client requests are replaced by a single one from Superfeedr. The service helps your content reach its intended audience more efficiently.

To prevent the Superfeedr bot from accessing your content feeds, you can add a disallow rule to your robots.txt file. Note that this will stop users of services that rely on Superfeedr from receiving your updates.

To block this bot, add the following lines to your robots.txt file:

User-agent: Superfeedr Bot
Disallow: /