7.7 Web Crawlers and Microsoft Content Management Server

---

Most organizations would like their public sites to be crawled and indexed by Web crawlers since this enables the site to be found from the popular search engines. Web crawlers generally work by taking the top-level pages submitted and then running through all of the links they can find.

Enduser search results are resolved from these stored pages or links.

Microsoft Content Management Server enables search engines to index any URLs that are accessible to guests visiting the site, and it is possible to add your own site search engine with access rights to index secure pages. Microsoft Content Management Server uses specific redirects, which may not be compatible with the Web crawler. If that is the case, HTML will need to be added to the template pages to ensure they are dealt with correctly.

---