Yahoo Slurp

**Anilrgowda** · 29-Dec-2006, 01:05 AM

ahoo! Slurp is Yahoo!'s web-indexing robot. The Yahoo! Slurp crawler collects documents from the Web to build a searchable index for search services using the Yahoo! search engine. These documents are discovered and crawled because other web pages contain links directing to these documents.
As part of the crawling effort, the Yahoo! Slurp crawler will take robots.txt standards into account to ensure we do not crawl and index those pages that you would not like to have returned via Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, it is neither considered for inclusion nor placed in the search engine's database.
Yahoo! Slurp follows HREF links. It does not follow SRC links. This means that Yahoo! Slurp does not retrieve or index individual frames referred to by SRC links.
Yahoo! Slurp has support for frames and makes an effort to crawl complex URLs such as those generated by forms, content generation systems, and dynamic page generation software. How can I prevent Yahoo Slurp from following links from a particular page or archiving a copy of a page?
Yahoo! Slurp obeys the noindex meta-tag. If you place:
<META NAME="robots" CONTENT="noindex">
-in the head of your web document, Yahoo! Slurp will retrieve the document, but it will not index the document or place it in the search engine's database.

<META NAME="robots" CONTENT="noindex"> Yahoo Slurp will retrieve the document, but it will not index the document. <META NAME="robots" CONTENT="nofollow"> Yahoo Slurp will not follow any links that are present on the page to other documents. <META NAME="robots" CONTENT="noarchive"> Yahoo maintains a cache of all the documents that we fetch, to permit our users to access the content that we indexed (in the event that the original host of the content is inaccessible, or the content has changed). If you do not wish us to archive a document from your site, you can place this tag in the head of the document, and Yahoo will not provide an archive copy(Cache) for the document.

Yahoo Slurp indexes not only the title and meta tags, but also the full text of webpages. So including quality content in the webpage is as important as including keywords in the title and meta tags.
Yahoo searches for pages and when it finds a page with the required keyword, it lists the page in its SERPs. The position of the page depends on the content. But there are chances for a page with the required keyword being left out due to poor content or because Yahoo could not find the page.

How to attract Yahoo slurp to crawl the site ?

There are 3 ways you attract the Yahoo Crawler in crawling the site:

1. Get links from sites that are regularly crawled by the Yahoo Robot, If that is done Yahoo regular visits the site and crawls it, Regular Yahoo visit is a good sign and helps a lot of getting good Ranking,

2. As yahoo says you can trigger the Yahoo Robot by browsing a site using the Yahoo companion toolbar, Yahoo says this will trigger the Yahoo slurp Bot,

3. Through the Infamous PFI/PPC program sitematch, This type of inclusion guarantee's an inclusion into the Yahoo index, so no problem using it,

29-Dec-2006, 01:05 AM	#1 (permalink)
Anilrgowda Administrator Posts: 18,704 Join Date: Jan 2006 Rep Power: 10 IM:	Yahoo Slurp ahoo! Slurp is Yahoo!'s web-indexing robot. The Yahoo! Slurp crawler collects documents from the Web to build a searchable index for search services using the Yahoo! search engine. These documents are discovered and crawled because other web pages contain links directing to these documents. As part of the crawling effort, the Yahoo! Slurp crawler will take robots.txt standards into account to ensure we do not crawl and index those pages that you would not like to have returned via Yahoo! Search Technology. If a page is disallowed to be crawled by robots.txt standards, it is neither considered for inclusion nor placed in the search engine's database. Yahoo! Slurp follows HREF links. It does not follow SRC links. This means that Yahoo! Slurp does not retrieve or index individual frames referred to by SRC links. Yahoo! Slurp has support for frames and makes an effort to crawl complex URLs such as those generated by forms, content generation systems, and dynamic page generation software. How can I prevent Yahoo Slurp from following links from a particular page or archiving a copy of a page? Yahoo! Slurp obeys the noindex meta-tag. If you place: <META NAME="robots" CONTENT="noindex"> -in the head of your web document, Yahoo! Slurp will retrieve the document, but it will not index the document or place it in the search engine's database. <META NAME="robots" CONTENT="noindex"> Yahoo Slurp will retrieve the document, but it will not index the document. <META NAME="robots" CONTENT="nofollow"> Yahoo Slurp will not follow any links that are present on the page to other documents. <META NAME="robots" CONTENT="noarchive"> Yahoo maintains a cache of all the documents that we fetch, to permit our users to access the content that we indexed (in the event that the original host of the content is inaccessible, or the content has changed). If you do not wish us to archive a document from your site, you can place this tag in the head of the document, and Yahoo will not provide an archive copy(Cache) for the document. Yahoo Slurp indexes not only the title and meta tags, but also the full text of webpages. So including quality content in the webpage is as important as including keywords in the title and meta tags. Yahoo searches for pages and when it finds a page with the required keyword, it lists the page in its SERPs. The position of the page depends on the content. But there are chances for a page with the required keyword being left out due to poor content or because Yahoo could not find the page. How to attract Yahoo slurp to crawl the site ? There are 3 ways you attract the Yahoo Crawler in crawling the site: 1. Get links from sites that are regularly crawled by the Yahoo Robot, If that is done Yahoo regular visits the site and crawls it, Regular Yahoo visit is a good sign and helps a lot of getting good Ranking, 2. As yahoo says you can trigger the Yahoo Robot by browsing a site using the Yahoo companion toolbar, Yahoo says this will trigger the Yahoo slurp Bot, 3. Through the Infamous PFI/PPC program sitematch, This type of inclusion guarantee's an inclusion into the Yahoo index, so no problem using it,

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)