Configuring the Crawler

This section describes how to configure the Crawler Configuration options and several sample use cases.


In this section:

Configuring the Crawler
Define the Groups of URLs to Crawl
Define Crawl Rules
Specify the File Name Extension and MIME Types to Crawl
Sample Configurations for Different Use Cases