Published on 2025-08-07T06:18:08Z

Offline Explorer bot

The Offline Explorer bot is not from a web service but is the user-agent for Offline Explorer, a desktop application that allows an individual to download a complete, browseable copy of a website. Its presence in your logs means a user has specifically targeted your site for offline viewing or archiving. The crawl is manually initiated and is for the user's private use, not for a public index.

What is the Offline Explorer bot?

The Offline Explorer bot is the user-agent for the desktop offline browser application Offline Explorer. This software allows a user to download an entire website to their local computer for offline viewing. It functions as a specialized web crawler that systematically downloads a site's content, preserving its navigation structure. The software identifies itself in server logs with a user-agent string like Offline Explorer/2.5. Unlike search engine crawlers, it downloads content exclusively for the individual user who initiated the download.

Why is the Offline Explorer bot crawling my site?

The Offline Explorer bot is crawling your site because an individual user has specifically chosen to download your website for offline access. This is not an automated, web-wide crawl. The frequency and scope of the crawling are determined entirely by how that user has configured their download project. The activity is initiated by an end user for personal reasons, such as research, archiving, or accessing your content in a location with limited internet connectivity.

What is the purpose of the Offline Explorer bot?

The purpose of the Offline Explorer bot is to support the Offline Explorer application, which allows users to access web content without an internet connection. Common uses include archiving content for research, creating local backups, or enabling browsing in areas with poor connectivity. The software creates a private, local copy of a website for the individual user and does not contribute to any centralized database. It offers no direct benefit to the website owner beyond allowing another user to access the content.

How do I block the Offline Explorer bot?

To prevent users from downloading your site with the Offline Explorer application, you can add a rule to your robots.txt file. This is the standard method for managing access for this type of tool.

To block the bot, add the following lines to your robots.txt file:

User-agent: Offline Explorer
Disallow: /

How to verify the authenticity of the user-agent operated by MetaProducts?

Reverse IP lookup technique

To verify user-agent authenticity, you can use host linux command two times with the IP address of the requester.

```
> host IPAddressOfRequest
```
This command returns the reverse lookup hostname (e.g., 4.4.8.8.in-addr.arpa.).

> host ReverseDNSFromTheOutputOfFirstRequest

If the output matches the original IP address and the domain is associated with a trusted operator (e.g., MetaProducts), the user-agent can be considered legitimate.

IP list lookup technique

Some operators provide a public list of IP addresses used by their crawlers. This list can be cross-referenced to verify a user-agent's authenticity. However, both operators and website owners may find it challenging to maintain an up-to-date list, so use this method with caution and in conjunction with other verification techniques.