Crawler Connector ArchitectureThe global connector architecture has two different server processes for hosting the connectors:
The main components of the Crawler architecture are:
Crawl RulesCrawl rules are a set of patterns, matching URLs and assigning actions to them. The most important rules are those which define whether a URL must be crawled, indexed, and whether to follow the links found.
Rule TypesWhen you specify a URL to crawl, you can define two main actions:
The index and follow rules have their negative counterparts: Don't index and Don't follow. An ignore rule can completely forbid a URL from being crawled. Note:
The URLs for which no crawl rules are specified, follow the default crawl rules defined in
General > Advanced options > Default behavior.
Rule PatternsMatching can use a wide set of patterns. The most common pattern uses a regular expression on the whole URL string. Other pattern types include many shortcuts, matching only a component of the URL, or comparing a length. Common patterns include:
The patterns available in the Administration Console are prefixes on entire URLs, supporting regular expressions. EscapingTo avoid escaping special characters like For example, to match all URLs on a site containing a RefreshBy default, no refresh is enabled in a new crawler. There are two modes to refresh crawled documents:
Recommendation:
Use the Smart refresh mode. Deterministic refresh is generally only
useful when crawling an intranet site with a controlled refresh rate, or when a specific
latency is required.
Smart RefreshIn Smart refresh mode, the crawler picks URLs from its repository depending on their age and update frequency. Fast-changing URLs are refreshed earlier than static ones. There are two additional parameters: Min and Max age:
For example, in the default configuration, a really fast-changing URL is never refreshed faster than every hour, and a completely static page is still refreshed every day. These parameters assume that the crawler has the available computing and network resources to achieve this refresh, else it will do at best. The Smart Refresh mode is therefore a good choice when you do know the update frequency of documents. Deterministic RefreshThe Deterministic refresh mode is available through a MAMI command
accessible by Note:
For more information about
cvcommand, see Get Started with Command-Line Interfaces in the Exalead CloudView Administration
Guide. The command is called
The command extracts all matching URLs from the crawler storage, and adds them in the fifos (the persistent structure containing the URLs to crawl). You can use the product scheduler to run this command periodically. Note:
This refresh can only be guaranteed if the crawler has enough computing and
network resources to crawl all refreshed URLs before the next refresh. The number of URLs
to refresh on a given host and the crawl delay (2.5 seconds by default) can also limit the
refresh rate.
In the following example, the command refreshes all pages that are older than a day on
./bin/cvcommand :$MAMIPORT /mami/crawl refreshDocuments crawlerName=$CRAWLER maxAgeS=86400 prefix=http://www.example.com/ Output in [2019/07/11-15:52:34.164] [info] [exa.bee.BeeJob 1127] [crawler.feedfetcher]Refreshing 8756 out of 10000 documents in prefix <http://www.example.com/> [2019/07/11-15:52:38.781] [info] [exa.bee.BeeJob 1127] [crawler.feedfetcher]Refreshing 17216 out of 20000 documents in prefix <http://www.example.com/> [2019/07/11-15:52:49.790] [info] [exa.bee.BeeJob 1127] [crawler.feedfetcher]Refreshing 25782 out of 30000 documents in prefix <http://www.example.com/> ... [2019/07/11-15:54:35.684] [info] [exa.bee.BeeJob 1127] [crawler.feedfetcher]Refreshed 145344 out of 172549 documents in prefix <http://www.example.com/> Note:
In this log, the last message means that the URLs have been scheduled for refresh but
are not necessarily refreshed yet.
The URLs to refresh are added to a fifo, by default, it uses the highest priority. See How Priorities Work. You can check the refresh progress in:
You can also run a deterministic refresh job periodically (at fixed times, dates or
intervals), by editing the In the following example, the You can specify several schedules by adding as many
<master:SchedulingConfig xmlns="exa:com.exalead.mercury.mami.master.v10"> <master:JobConfigGroup name="refresh_crawlers"> <master:DispatchJobConfig name="refresh_crawler"> <master:DispatchMessage xmlns="exa:exa.bee" messageName="refreshDocuments" serviceName="/mami/crawl"> <master:messageContent> <master:KeyValue key="crawlerName" value="CRAWLERNAME" /> <master:KeyValue key="maxAgeS" value="3600" /> <!-- optional, don't refresh documents younger than 1 hour, else all documents regardless of age --> <master:KeyValue key="prefix" value="URLPREFIX" /> <!-- optional, only refresh documents with URL beginning with prefix, else all documents --> </master:messageContent> </master:DispatchMessage> </master:DispatchJobConfig> </master:JobConfigGroup> <master:TriggerConfigGroup name="exa"> <master:CronTriggerConfig cronExpression="0 0 0 * * ?" misfireInstruction="do_nothing" jobName="refresh_crawler" jobGroupName="refresh_crawlers" name="refresh_crawlers" calendarName="dummy_calendar" /> </master:TriggerConfigGroup> </master:SchedulingConfig> About Site CollapsingOften a site does not have a lot of rich content about a specific topic. If a Exalead CloudView user searches for that topic, the search results are overwhelmed with many pages from that site. The result is that other sites with important relevant information get crowded out. The idea of site collapsing is to treat all the pages from a site as a single page. In this case, Exalead CloudView tries to select the most relevant page from the site, and only shows that one page in the search results. Other sites appear higher in the search results. Site collapsing works on the assumption that if the page selected to represent the site does not contain the content a user is looking for, there must be enough good links (navigation menus) within the site to allow users to quickly find what they are looking for. By default, the site collapsing ID is based on the host part of the URL. All URLs belonging to
the same host share the same ID and are shown as one collapsed result. You can configure the
site ID to include more segments of the URL's path to distinguish sites on the same host.
You can do so by specifying the recursion with the Depth parameter.
By default, the value is This option pushes a meta called site_id with the documents to index. This meta is based on the site extracted from a URL. To enable site collapsing, you also need to create a group in Search Logics > Sort & Relevance and in the Expression field, enter site_id. URL Processing URL Handling of Crawled Sites
The Crawler connector has the following URL handling methods or constraints:
| |||||||||||
