Crawler

Crawler: Automatically scrape web content and import it into the Knowledge Base

What Is This?

The crawler automatically scrapes content from web pages and imports it into the Knowledge Base. Simply provide a URL, and it will crawl the text content of that page (or the entire website) and convert it into knowledge that the Agent can reference.

When Do You Need It?

• Website content: Automatically import product descriptions and service information from your company website

• Public information: Scrape publicly available web pages such as regulations, announcements, and technical documents

• Continuous updates: When website content changes, periodically re-crawl to keep the Knowledge Base up to date

Relationship Between Crawler and Knowledge Base

The crawler is one of the data sources for the Knowledge Base. Its role is:

Data Source          Knowledge Base      Agent
─────────           ──────              ──────
Manual file upload ──→
Create FAQ         ──→  Enterprise       ──→  Look up and respond
Crawler scraping   ──→  Knowledge Base

The content scraped by the crawler ultimately enters the Knowledge Base. The Agent does not interact with the crawler directly.

What Do I Need to Do?

1. Enter a URL — Tell the crawler which web page or website to scrape

2. Set the scope — Scrape a single page only, or recursively crawl subpages

3. Run the crawl — Start the crawler and wait for it to complete

4. Import into Knowledge Base — Import the crawl results into a specified Knowledge Base

5. Verify the content — Check whether the imported content is correct and complete

Further Reading

• How to Use the Crawler (Scrape Data) Feature