Debug Authentication Problems

This task shows you how to:

See Also
Fetcher Authentication Process
Troubleshooting the Crawler Connector

Look for Traces in the Log

  1. Set the logger fetcher to the trace level:
    1. Edit <DATADIR>/config/Logging.xml
    2. Add the line: 

      <master:LoggingLevel level="trace" process="crawler-exa0"/>
      Note: process is the process name of your crawler.

  2. Click Apply.
  3. Look for full HTTP traces in the crawler process log.
    Recommendation: This makes the crawler process log really verbose. Do not enable it continuously.

    You can also use HTTPFox (Firefox extensions), Wireshark, or similar tools to analyze the HTTP requests during the authentication process, and compare behaviors in a browser and in the fetcher.

    If everything fails, you can still try to log in with a browser, export the cookies, and give them to the fetcher. This only works until the session expires, but it is better than nothing.

Convert Cookies Files

If everything fails, you can still try to log in with a browser, export the cookies, and give them to the fetcher. This only works until the session expires. For example, with Firefox.

  1. Export the cookies.
  2. Convert the cookies file with: 

    sed -e
's#\([^\t]*\)\t\([^\t]*\)\t\([^\t]*\)\t\([^\t]*\)\t\([^\t]*\)\t\([^\t]*\)\t\([^\t\n\r]*\)#<HTTPCookie
xmlns="exa:exa.net" version="1" name="\6" value="\7" path="\3" domain="\1"
secure="false" expires="2099/04/09-11:09:32"/>#' cookies.txt (change the expiry time if necessary).

  3. Stop the crawler process.
  4. Overwrite the content of the <DATADIR>/fetcher/crawler-INSTANCE/fetcher_CRAWLERNAME/cookies.txt file with the converted cookies.txt file.
  5. Start the crawler process.