This section describes how to configure the Crawler Configuration options and several sample use cases.
Configuring the Crawler Define the Groups of URLs to Crawl Define Crawl Rules Specify the File Name Extension and MIME Types to Crawl Sample Configurations for Different Use Cases