Use the Crawler (Data Scraping) Feature
Enter a URL, set the crawl scope, and choose a one-time run or recurring schedule. MaiAgent will automatically crawl the website and import its pages into a knowledge base, where you can track the res
Purpose and Benefits
As a business user, you may often be asked by a manager to consult or compile regulations published on websites, or to keep an AI assistant up to date with the latest product information and FAQs on your company's website.
If you have a technical background or help from an engineer, you might automate data collection with a crawler. For nontechnical users, however, collecting pages by hand takes time and effort and can easily miss important information.
MaiAgent's No-Code web scraping feature lets you:
Crawl an entire website scope at once: Set the crawl depth, page limit, and URL rules to choose which pages to crawl.
Run once or on a recurring schedule: Crawl immediately, or recrawl automatically every day, week, or month.
Import automatically into a knowledge base: Import crawled pages directly into a selected knowledge base. For URLs that already exist, choose to skip, overwrite, or allow duplicates.
Track import results: See how many pages succeeded or failed in each import record.
Use Case
A customer service manager wants her AI support assistant's answers to stay consistent with the FAQs on the company website. She creates a knowledge base for those FAQs, then opens Crawler and creates a crawl request. She enters the website's FAQ URL and sets a low maximum crawl depth and page limit so she can first confirm that only FAQ pages will be crawled. She chooses Scheduled run for every Monday at 9 a.m., selects the new knowledge base as the automatic import destination, and chooses Skip when a URL already exists. The weekly synchronization task then appears on the Schedules tab. After each crawl, she can open Import history in the crawl request details to see which knowledge base received the pages and how many succeeded or failed. When the website's FAQs change, she no longer needs an engineer to compile the content again.
How Do I Crawl a Website?
1. Open the Crawler Page
In the left menu, click AI Features → Crawler. The page has two tabs:
Page crawl requests: Lists every crawl, including one-time runs and crawls started automatically by schedules. Search by name or URL, and filter by type (one-time/scheduled) and crawl status (completed/in progress/failed).
Schedules: Lists all recurring crawl schedules.
2. Enter a Task Name and URL
Click Create page crawl request in the upper right. The setup flow has three steps: URL, Crawl settings, and Execution method.
In the first step, enter:
Task name: A name that helps you identify this crawl, such as “Daily product page crawl.”
URL: The URL where crawling should start.
Then click Next.
3. Set the Crawl Depth and Scope
The second step, Crawl settings, controls how many pages to crawl. Basic settings has two fields:
Maximum crawl depth
Number of link levels to follow from the starting URL. Choose 1–10; the default is 2.
Maximum pages to crawl
Maximum number of pages in this crawl. Choose 10, 50, 100, 500, 1000, or 5000; the default is 100.
For more precise control, expand Advanced settings:
URL matching rules
Crawl only URLs that match a rule, such as /products/*. Without rules, all pages within the crawl scope are included.
URL exclusion rules
Do not crawl URLs that match a rule, such as /admin/* or \.pdf$.
Boundary settings
Choose whether to allow crawling same-level or parent paths, include subdomains, or allow external links. All are off by default.
Sitemap strategy
Prefer sitemap (default), Ignore sitemap, or Use sitemap only.
Content extraction settings
Extract main content only and Ignore URL parameters are on by default. You can also set the JS rendering wait time to 0, 500, 1000, or 2000 ms.
Enabling Allow external links crawls URLs outside your website and may greatly increase the number of pages and the cost. Start with a small depth and page limit to confirm the scope before increasing them.
When you finish, click Next.
4. Choose an Execution Method and Knowledge Base
The third step, Execution method, determines when to crawl and where to import the results.
Execution method
One-time run: Start one crawl immediately after creating the request.
Scheduled run: Crawl according to a recurring schedule (see Set Up a Recurring Schedule in the next section).
Knowledge base settings
Import into an existing knowledge base: Under Automatic import destination (optional), select one or more knowledge bases. Pages are imported automatically after crawling. Leave this empty to crawl without importing; you can import manually later.
Automatically create a new knowledge base: Create a new knowledge base named after the task on each run and import the crawled content into it.
After selecting an import destination, you can choose How to handle existing URLs for URLs already imported into that knowledge base:
Skip (default)
Skip URLs already imported into this knowledge base.
Overwrite
Replace the existing file with newly crawled content.
Allow duplicates
Create a new file even if the URL already exists.
5. Preview and Create
Before creating the request, click Preview to check which pages will be crawled. The preview explores website URLs and incurs a charge. A confirmation dialog appears first; click Confirm to proceed.
The preview shows the number of URLs found, URLs to crawl and exclude, estimated pages and time, and a list of pages to crawl.
If everything looks correct, click Create. A one-time request appears on the Page crawl requests tab and starts crawling.
If the URL was crawled before, a This URL has been crawled before confirmation dialog shows the previous crawl time, page count, and imported knowledge bases. To crawl it again, click Crawl again; otherwise, click Cancel.
Set Up a Recurring Schedule
When website content changes regularly, choose Scheduled run in step 3 to recrawl and import into a knowledge base automatically.
Frequency
Once, Daily, Weekly, or Monthly.
Time zone
Time zone used for the schedule.
Start date / End date
Period when the schedule is active. End date is optional; a one-time frequency only requires a start date.
Execution time
Time of each run.
Days of the week
For weekly schedules, select the days to run (multiple selections allowed).
Day of the month
For monthly schedules, enter a day from 1–31.
After creation, the schedule appears on the Schedules tab with its settings and status (Active / Paused). The action icons let you:
Edit: Open Edit schedule to change the task name, crawl settings, schedule time, and knowledge base settings. The starting URL cannot be changed after creation.
Execution history: View the time and crawl status of each run and open its details.
Pause / Enable: Temporarily pause or re-enable the schedule.
Delete: Delete the schedule.
Each scheduled run crawls pages and consumes Credits. Pause the schedule when you temporarily do not need synchronization.
View Crawl Results and Import History
On the Page crawl requests tab, click Details for a request to open the Crawl request details page:
Crawl progress: Number of pages processed and total pages.
Crawl task settings: Expand to review the URL, crawl settings, and execution method chosen at creation.
Crawled pages: Titles, URLs, crawl statuses, and completion times of the pages crawled.
Switch to the Import history tab to see each import's time, knowledge base, Import method (automatically imported pages show Automatic), and counts of successful and failed pages.
Click the action icon to open the Imported pages list and review each page's import status and corresponding knowledge base file.
On the Page crawl requests list, you can also click View on a completed request to open Failed import records. Select a knowledge base to see which pages failed to import and why.
Imported content appears as .md files in the knowledge base's document list. You can set tags and metadata just as you can for other files. The Source column shows a Crawler tag; click it to return to the corresponding crawl request details.
Manually Import Crawled Pages
If you did not set up automatic import or want to import pages into another knowledge base, import them manually:
On the crawl request details page, click Import into knowledge base (or click Import for the request in the list).
Select the target Knowledge base and How to handle existing URLs.
Select the pages to import, then click Import.
If you choose Skip and some URLs have already been imported, the system reports how many import tasks were created and how many duplicate URLs were skipped.
A manual import recrawls the latest page data and incurs a crawling charge.
Web Scraping Considerations
Make sure you have permission to crawl the target website's content.
Crawling, previewing, and manually importing all consume Credits. Start with a small maximum crawl depth and page limit to verify the scope before increasing them.
After crawling, use Search Test to check data quality.
If website content changes often, use a recurring schedule to keep the information current, and choose Skip or Overwrite to avoid duplicate files.
Last updated
Was this helpful?