MetaInspector bot

MetaInspector is a web scraping tool, specifically a Ruby gem, used by developers to extract metadata from web pages. It is not a single bot but a library that can be incorporated into many different applications. It is designed to parse a web page's HTML to extract its title, meta tags, links, images, and other structured content. When an application using this library visits a website, it often identifies itself with a user-agent string like MetaInspector/ [version] (+https://github.com/jaimeiniesta/metainspector), which transparently points to the library's GitHub page.

If you see a MetaInspector user-agent in your site logs, it means a third-party application built with this library is accessing your content. Unlike large-scale search crawlers, its visits are typically targeted and initiated by a specific application's need to extract information from your pages. This could be for generating link previews, monitoring content for changes, or aggregating content for a curation service. The frequency of visits depends entirely on how the third-party application is configured.

MetaInspector serves as a tool for applications that need to process web page information. Common uses include generating link previews for social media or messaging apps, monitoring websites for content changes, and building content curation services. The data collected is used within the specific application that implemented the library. The value to a website owner varies; if the scraping leads to increased visibility through content sharing, it can be beneficial. However, it can also consume server resources with no direct benefit in return.

To prevent applications using the MetaInspector library from scraping your site, you can add a disallow rule to your robots.txt file. This is the standard method for managing access for web scraping tools that identify themselves.

Add the following lines to your robots.txt file to block MetaInspector:

User-agent: MetaInspector
Disallow: /