Embedly

Embedly is a set of front-end developer tools that functions as a middleware service for generating rich content embeds and previews. It is a content fetcher, not a traditional web crawler, meaning it only accesses URLs that are explicitly provided by an end user through a client application. When making a request, it identifies itself with the user-agent string Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/). It supports over 200 content providers and uses its API to return standardized metadata for creating previews.

Embedly is accessing your website because a user has shared a link to your content on a platform that utilizes Embedly's services (e.g., a social media site or a content management system). It does not proactively crawl your site. Instead, it is triggered by a specific user action to fetch metadata—such as titles, descriptions, and thumbnails—to generate a rich preview for the shared link. The frequency of visits is entirely dependent on how often your content is shared on these platforms.

The core purpose of Embedly is to improve the user experience on other platforms by transforming plain URLs into rich, interactive content previews. When a link is shared, Embedly retrieves information about the URL, checks it for malware, extracts content, and returns standardized metadata that the embedding platform can use. For website owners, this service provides a significant benefit by making your shared links more visually appealing and interactive, which can drive more traffic to your site and improve user engagement with your content.

If you wish to prevent Embedly from generating previews of your content, you can block it using your robots.txt file. Note that doing so will cause links to your site to appear as plain text URLs when shared on platforms that use this service.

To block Embedly, add the following lines to your robots.txt file:

User-agent: Embedly
Disallow: /