> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/km/scrape-website.md).

# How to Use the Crawler (Web Scraping) Feature

## Feature Purpose and Value <a href="#feature-purpose-and-value" id="feature-purpose-and-value"></a>

As a business professional, you may often receive instructions from management to reference or compile regulatory data from certain public websites.

If you have a technical background or engineering support, you might be able to write a scraping script to automatically extract data. However, for non-technical users, the only option is usually to manually organize pages one by one — a time-consuming, labor-intensive process that is also prone to missing key information.

In such cases, you can leverage MaiAgent's web scraping feature to quickly extract website content through a No-Code approach, automatically creating structured data. This significantly improves information organization efficiency, allowing you to invest your time in higher-value core business tasks.

## How to Scrape? <a href="#how-to-scrape" id="how-to-scrape"></a>

To create a scraping request, you can:

1. Create a Page Scraping Request

Go to the left sidebar "<mark style="color:blue;">AI Features</mark>" → "<mark style="color:blue;">Web Scraping</mark>", and click the "<mark style="color:blue;">+ Create Page Scraping Request</mark>" button in the upper right corner.

2. Enter the URL

Enter the URL of the page you want to scrape, and click the \[<mark style="color:blue;">Confirm</mark>] button.

{% hint style="warning" %}

* Please note that the URL cannot exceed 200 characters
* If the status has not changed, you can click the refresh button in the upper right corner to update the status
  {% endhint %}

<div><figure><img src="/files/XkFmY6nhSUVxf7XipPNW" alt=""><figcaption></figcaption></figure> <figure><img src="/files/fpSajfNH0QoncOmru1Nb" alt=""><figcaption></figcaption></figure></div>

3. Review Scraped Data

Once the status shows as completed, click "Import" on the right side to review the scraped data entries.

4. Select Data

Check the boxes on the left to select the data you want to import into the knowledge base. After selecting, click the "<mark style="color:blue;">Import</mark>" button, and the data will be automatically imported into the AI assistant's knowledge base.

{% hint style="info" %}
To browse more data entries on the same page, you can click "<mark style="color:blue;">10 entries/page</mark>" in the lower right corner to expand the viewing range.
{% endhint %}

<div><figure><img src="/files/q9tZG5q0EBq13j3Ya8IJ" alt=""><figcaption></figcaption></figure> <figure><img src="/files/29SNzQK0VAuqPoP7sqSk" alt=""><figcaption></figcaption></figure></div>

In the knowledge base, you can see the data presented as .md files, and just like regular data, you can configure tags and metadata for them.

<figure><img src="/files/AESLdbRuykznnYwM1QKC" alt=""><figcaption></figcaption></figure>

## Web Scraping Considerations <a href="#scraping-considerations" id="scraping-considerations"></a>

* Ensure you have permission to scrape the target website's content
* It is recommended to test with a small range of data before performing large-scale scraping
* After scraping is complete, you can verify data quality through the [Search Testing feature](/maiagent-user-guide/maiagent-user-guide-en/km/test-search-result.md)
* Regularly update scraped data to maintain information timeliness


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/km/scrape-website.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.