About the HTTP Fetcher

The HTTP Fetcher is the component allowing to download a document from its URL.

This page discusses:

HTTP Common Parameters
Conditional Fetch
HTTP Authentication
HTML Form Authentication

HTTP Common Parameters

Cookies are always processed, but never stored by default. They are however automatically enabled, stored, and sent back on sites configured with an HTML form authentication. The global Enable cookies for all sites parameter allows you to store and send cookies for all sites. Only cookies following the standard privacy rules are stored, as web browsers do.

The User agent by default identifies itself as a Mozilla-compatible Exalead CloudView. When crawling the public internet, customize it with a URL pointing to a page describing the crawler and how to contact the administrator.

The From field is empty by default, and can contain a contact email address, which is useful when crawling the public internet.

You can add any HTTP header to the default request. The default configuration only contains Accept: */*, which is required by some sites.

Conditional Fetch

The fetcher uses the Last-Modified date and E-Tag HTTP Headers to allow conditional fetch.

The crawler uses this condition when refreshing pages, to avoid fetching unchanged documents. In that case, the server only answers 304 Not Modified.

HTTP Authentication

The fetcher can access pages protected through different kind of authentication protocols:

Basic only requires a username and a password. They are sent as plain text in the HTTP request.
Digest requires a username, a password, and an optional realm. Unlike Basic, the Digest protocol avoids sending the password as plain text.
NTLM requires a username, a password, and either a hostname or a domain.

Configure URL patterns to tell the fetcher which sites the login information applies to. They have the same syntax and meaning that the patterns used in the crawler configuration. They must match only the URLs where the login information is required.

Important: If you do not configure any URL pattern, the authentication applies to all URLs crawled. This can be dangerous, as the password might be sent to a third party.

HTML Form Authentication

The fetcher also supports authentication using HTML forms. This applies to any website where the login procedure involves filling in an HTML form and submitting it through a POST request.

What You Must Know

To properly configure the fetcher to authenticate using an HTML form, gather the following information:

Find patterns to identify which URLs require authentication. Sometimes authentication is not required or not relevant to retrieve some parts of a website, for example static resources or images. URL patterns are the same than for standard HTTP authentication (see above), and the same recommendations apply. They work the same way as crawl rule patterns, and support the same syntax.
Describe how to identify a successfully authenticated page from an unauthenticated one. This authentication test requires a success/failure condition, and is applied to all crawled URLs matching the above patterns. You can either choose a characteristic of the failed authentication page, to make a "failure condition", or a characteristic of a successfully authenticated page, to make a "success condition". See below for examples.
Find the following information to authenticate:
- First, find the URL containing the authentication form. This URL is called the gateway URL. It must be unique for the whole site. It is often the URL you are redirected to, when attempting to load a page without being authenticated.
- Then, find the authentication form in the HTML. Check, using "view source", that the form is statically embedded in the HTML and not dynamically generated using javascript. In that last case, the fetcher cannot find it. If it is in an iframe, use the iframe URL instead. If the form is not the first one in the page, tell the fetcher how to find the right one: you can use the element's name, id, or class as required. The first matching form is used.
- To specify the form input in the fetcher Form fields, you have to find the input name of each required parameter, usually a user name, and a password. In the following input form example, the user field contains the user name, and the passwd field contains the password.
```
<form method="post" action="/login" id="login_login-main" class="login-form login-formside">
  <input type="hidden" name="op" value="login-main" />  
  <input name="user" placeholder="user name" type="text" />
  <input name="passwd" placeholder="user password" type="password" />
  <div id="remember-me">
    <input type="checkbox" name="rem" id="rem-login-main" />
    <label for="rem-loginmain">Remember me</label>
    <a class="recover-password" href="/password">Reset password</a>
  </div>
  <div class="submit">
    <button class="btn" type="submit">Connect</button>
  </div>
</form>
```
- Also check that the login process does not require javascript. In the example above, the action is a relative URL: /login. This is the destination of the POST login request. If it is absent, and the post URL is generated by a javascript method like onSubmit='validateForm();', it is unlikely the fetcher can submit it.

Write a Correct Success/Failure Condition

A good condition must return:

Failure if and only if the fetch result shows that you are not authenticated.
Success if and only if the fetch result shows that you have successfully authenticated.


Body text match condition	- Finding a string like `Welcome $USER` in the content of the page may be a good success criteria. - Finding a string like `Please authenticate to access this document` would be a good failure test.
Redirection condition	- The fetcher tests only if a page contains a redirection to another page and does not follow the redirection (a page and its redirection are two different documents). - Many sites redirect unauthenticated requests to a login page. Assuming the first redirection destination is `/login/login.html`, a good condition would be Failure if redirection matches `/login/login/html`. Matching only the presence of a redirection is usually a bad idea. Sites often contain lots of invisible redirections, even when authenticated.

Limitations: login procedures requiring javascript are not supported, except in the simple case of auto-submitted forms (usually used to transfer cookies or session IDs across domains without setting large query parameters).

Common Misconceptions

Authentication process and authentication test (the Success/Failure condition) are two independent processes. The authentication test condition is tested every time a document is fetched, but the authentication itself is executed only when the condition returns a failure.
The crawler and the fetcher are independent. The crawler does not know about authentication. The process is transparent. The fetcher is like a black box, accepting URLs as input and returning authenticated documents. That means that login pages must never be indexed. If it happens, it means that your authentication process is wrong: either the authentication process has failed and the fetcher is not authenticated, or the condition always returns success erroneously (and never triggers the authentication).
Cookies are automatically enabled and handled, there is no need to enable them manually. Use the Enable cookies for all sites option only if an unauthenticated site requires them.

Cookies are handled transparently by the fetcher. It behaves as much as possible like a standard web browser.