- if the url is http://foo.com/bar it
crawls only pages whose urls begin with http://foo.com/bar -
if the url is http://foo.com it crawls only urls beginning
with http://foo.com/ OR http://www.foo.com/
Note:
If the home page is a redirection or a meta redirection to another site,
it follows it and crawls the destination site. Except for the home page, it
does not follow redirections to pages external to the site.
|