Published on 2025-08-07T06:18:08Z

AwarioBot

AwarioBot is the official web crawler for Awario, a social media and web monitoring platform. Its purpose is to scan public web pages, blogs, and forums to find mentions of specific keywords, brands, or products that Awario's customers are tracking. It operates with a 'gentle' crawl behavior to minimize impact on website performance. The data it collects powers Awario's brand monitoring and sentiment analysis services.

What is AwarioBot?

AwarioBot is a legitimate web crawler operated by the web and social media monitoring platform, Awario. Its function is to discover and collect new or updated public content from websites, blogs, and forums. The bot is part of a family of crawlers, including AwarioSmartBot and AwarioRssBot, that work together to track brand mentions and online conversations for Awario's clients. It identifies itself clearly in server logs with user-agent strings like Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html). A key characteristic is its polite crawling behavior, designed to make requests slowly (one per three seconds, or less if specified in robots.txt) to minimize server impact.

Why is AwarioBot crawling my site?

AwarioBot is visiting your website because your public content likely contains mentions of brands, products, or topics that Awario's customers are monitoring. The bot scans for specific keywords and phrases but does not target sensitive or private information. It typically visits a page only once for analysis, unless the content is updated and needs to be refreshed in Awario's database. The frequency of its visits depends on how often your site publishes new content that matches the monitoring queries set by Awario users.

What is the purpose of AwarioBot?

The primary purpose of AwarioBot is to collect the data that powers Awario's social listening and brand monitoring service. This service helps businesses track their online presence, analyze customer sentiment, manage their reputation, and identify potential sales leads. The content collected by the bot is fed into Awario's analytics platform, providing valuable insights for marketers and brand managers. While the service primarily benefits Awario's customers, website owners may see indirect benefits if their content is discovered by brands seeking industry insights or potential partners.

How do I block AwarioBot?

If you prefer that AwarioBot and its related crawlers do not access your site, you can add rules to your robots.txt file. This file gives instructions to well-behaved bots.

To block all Awario crawlers, add the following directives to your robots.txt file:

User-agent: AwarioBot
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: AwarioRssBot
Disallow: /

How to verify the authenticity of the user-agent operated by Awario?

Reverse IP lookup technique

To verify user-agent authenticity, you can use host linux command two times with the IP address of the requester.

```
> host IPAddressOfRequest
```
This command returns the reverse lookup hostname (e.g., 4.4.8.8.in-addr.arpa.).

> host ReverseDNSFromTheOutputOfFirstRequest

If the output matches the original IP address and the domain is associated with a trusted operator (e.g., Awario), the user-agent can be considered legitimate.

IP list lookup technique

Some operators provide a public list of IP addresses used by their crawlers. This list can be cross-referenced to verify a user-agent's authenticity. However, both operators and website owners may find it challenging to maintain an up-to-date list, so use this method with caution and in conjunction with other verification techniques.