PanguBot

PanguBot is a web crawler operated by Huawei that serves as a data acquisition tool for its PanGu large language model (LLM), a multimodal AI system that can process text, images, and other data. As an AI data scraper, PanguBot's purpose is to download publicly available web content to be used for training Huawei's AI models. The bot identifies itself with the standardized user-agent string PanguBot. Its architecture is optimized for collecting diverse training data, and it shows a preference for multimedia content and sites with high information density.

PanguBot is visiting your website to collect training data for Huawei's PanGu AI model. It looks for a wide range of content, with a particular interest in text, images, and technical documentation. The frequency of its visits depends on your site's content quality and update schedule. This crawling is part of Huawei's broader effort to improve its AI systems and, like many AI data scrapers, it accesses publicly available content that is freely accessible by default.

The purpose of PanguBot is to support Huawei's PanGu LLM by collecting diverse training data from the web. The content it gathers helps improve the AI model's capabilities in natural language processing and image recognition. For website owners, there is no direct benefit from the bot's crawling activity; its purpose is solely to improve Huawei's AI systems. This has led to concerns from some content creators about their work being used to train commercial AI systems without permission or compensation.

To prevent PanguBot from using your website's content to train its AI models, you can add a disallow rule for it in your robots.txt file. This is the standard method for managing crawler access.

Add the following lines to your robots.txt file to block PanguBot:

User-agent: PanguBot
Disallow: /