Crawler

What Is This?

The crawler automatically scrapes content from web pages and imports it into the Knowledge Base. Simply provide a URL, and it will crawl the text content of that page (or the entire website) and convert it into knowledge that the Agent can reference.

When Do You Need It?

• Website content: Automatically import product descriptions and service information from your company website

• Public information: Scrape publicly available web pages such as regulations, announcements, and technical documents

• Continuous updates: When website content changes, periodically re-crawl to keep the Knowledge Base up to date

Relationship Between Crawler and Knowledge Base

The crawler is one of the data sources for the Knowledge Base. Its role is:

Data Source          Knowledge Base      Agent
─────────           ──────              ──────
Manual file upload ──→
Create FAQ         ──→  Enterprise       ──→  Look up and respond
Crawler scraping   ──→  Knowledge Base

The content scraped by the crawler ultimately enters the Knowledge Base. The Agent does not interact with the crawler directly.

What Do I Need to Do?

1. Enter a URL — Tell the crawler which web page or website to scrape

2. Set the scope — Scrape a single page only, or recursively crawl subpages

3. Run the crawl — Start the crawler and wait for it to complete

4. Import into Knowledge Base — Import the crawl results into a specified Knowledge Base

5. Verify the content — Check whether the imported content is correct and complete

Further Reading

• How to Use the Crawler (Scrape Data) Feature