How to Use the Crawler (Web Scraping) Feature

Simply enter a URL, and MaiAgent will structurally scrape the text and link data from the page for you, making it easy to quickly select data to import into the knowledge base and build your AI assist

Feature Purpose and Value

As a business professional, you may often receive instructions from management to reference or compile regulatory data from certain public websites.

If you have a technical background or engineering support, you might be able to write a scraping script to automatically extract data. However, for non-technical users, the only option is usually to manually organize pages one by one — a time-consuming, labor-intensive process that is also prone to missing key information.

In such cases, you can leverage MaiAgent's web scraping feature to quickly extract website content through a No-Code approach, automatically creating structured data. This significantly improves information organization efficiency, allowing you to invest your time in higher-value core business tasks.

How to Scrape?

To create a scraping request, you can:

1. Create a Page Scraping Request

Go to the left sidebar "AI Features" → "Web Scraping", and click the "+ Create Page Scraping Request" button in the upper right corner.

1. Enter the URL

Enter the URL of the page you want to scrape, and click the [Confirm] button.

• Please note that the URL cannot exceed 200 characters

• If the status has not changed, you can click the refresh button in the upper right corner to update the status

1. Review Scraped Data

Once the status shows as completed, click "Import" on the right side to review the scraped data entries.

1. Select Data

Check the boxes on the left to select the data you want to import into the knowledge base. After selecting, click the "Import" button, and the data will be automatically imported into the AI assistant's knowledge base.

To browse more data entries on the same page, you can click "10 entries/page" in the lower right corner to expand the viewing range.

In the knowledge base, you can see the data presented as .md files, and just like regular data, you can configure tags and metadata for them.

Web Scraping Considerations

• Ensure you have permission to scrape the target website's content

• It is recommended to test with a small range of data before performing large-scale scraping

• After scraping is complete, you can verify data quality through the Search Testing feature

• Regularly update scraped data to maintain information timeliness