Using Robots.txt To Prevent Search Indexing

**Anilrgowda** · 30-Jan-2007, 04:19 AM

Sometimes there are parts of your website you don't want accessed by the search engines - for any number of reasons, like sensitive private data, articles that require subscriptions - whatever.Google recently posted a tutorial about how to use the robots.txt file to block search engines from indexing specific parts of the site.

Though we've talked about robots.txt at length, it often good to remember the newbies (hi, newbies!). So it was good timing that this tutorial was posted, even if product manager Dan Crow apologetically butchered up a certain brilliant paragraph by Douglas Adams in the process.

He redeems himself by providing the following easy to decode example of how to implement one of these necessary files:

Here is a simple example of a robots.txt file.

User-Agent: Googlebot
Disallow: /logs

The User-Agent line specifies that the next section is a set of instructions just for the Googlebot. All the major search engines read and obey the instructions you put in robots.txt, and you can specify different rules for different search engines if you want to. The Disallow line tells Googlebot not to access files in the logs sub-directory of your site. The contents of the pages you put into the logs directory will not show up in Google search results.

Preventing access to a file

If you have a news article on your site that is only accessible by registered users, you'll want it excluded from Google's results. To do this, simply add a META tag into the html file, so it starts something like:

<html>
<head>
<meta name="googlebot" content="noindex">

This stops Google from indexing this file. META tags are particularly useful if you have permission to edit the individual files but not the site-wide robots.txt. They also allow you to specify complex access-control policies on a page-by-page basis.

**Gracia** · 13-Apr-2007, 02:32 AM

am curious to see some example uses and implementations of robots.txt.. specifically implementations and the reasons behind them for increasing SEO. The robots.txt I am using is based on the example: SEO with robots.txt

WordPress 2.1 robots.txt

User-agent: * # disallow all files in these directoriesDisallow: /cgi-bin/Disallow: /admin/Disallow: /comments/Disallow: /z/j/Disallow: /z/c/Disallow: /about/legal-notice/Disallow: /about/copyright-policy/Disallow: /about/terms-and-conditions/Disallow: /about/feed/Disallow: /about/trackback/Disallow: /contact/Disallow: /stats*Disallow: /tagDisallow: /category/uncategorized*# disallow all files ending with these extensionsDisallow: /*.php$Disallow: /*.js$Disallow: /*.inc$Disallow: /*.css$Disallow: /*.txt$# disallow all files in /wp- directorysDisallow: /wp-*/# disallow all files with? in urlDisallow: /*?

30-Jan-2007, 04:19 AM	#1 (permalink)
Anilrgowda Administrator Posts: 18,715 Join Date: Jan 2006 Rep Power: 10 IM:	Using Robots.txt To Prevent Search Indexing Sometimes there are parts of your website you don't want accessed by the search engines - for any number of reasons, like sensitive private data, articles that require subscriptions - whatever.Google recently posted a tutorial about how to use the robots.txt file to block search engines from indexing specific parts of the site. Though we've talked about robots.txt at length, it often good to remember the newbies (hi, newbies!). So it was good timing that this tutorial was posted, even if product manager Dan Crow apologetically butchered up a certain brilliant paragraph by Douglas Adams in the process. He redeems himself by providing the following easy to decode example of how to implement one of these necessary files: Here is a simple example of a robots.txt file. User-Agent: Googlebot Disallow: /logs The User-Agent line specifies that the next section is a set of instructions just for the Googlebot. All the major search engines read and obey the instructions you put in robots.txt, and you can specify different rules for different search engines if you want to. The Disallow line tells Googlebot not to access files in the logs sub-directory of your site. The contents of the pages you put into the logs directory will not show up in Google search results. Preventing access to a file If you have a news article on your site that is only accessible by registered users, you'll want it excluded from Google's results. To do this, simply add a META tag into the html file, so it starts something like: <html> <head> <meta name="googlebot" content="noindex"> This stops Google from indexing this file. META tags are particularly useful if you have permission to edit the individual files but not the site-wide robots.txt. They also allow you to specify complex access-control policies on a page-by-page basis.

13-Apr-2007, 02:32 AM	#2 (permalink)
Gracia Fix my Error! Posts: 18 Join Date: Apr 2007 Rep Power: 0 IM:	Re: Using Robots.txt To Prevent Search Indexing am curious to see some example uses and implementations of robots.txt.. specifically implementations and the reasons behind them for increasing SEO. The robots.txt I am using is based on the example: SEO with robots.txt WordPress 2.1 robots.txt User-agent: * # disallow all files in these directoriesDisallow: /cgi-bin/Disallow: /admin/Disallow: /comments/Disallow: /z/j/Disallow: /z/c/Disallow: /about/legal-notice/Disallow: /about/copyright-policy/Disallow: /about/terms-and-conditions/Disallow: /about/feed/Disallow: /about/trackback/Disallow: /contact/Disallow: /statsDisallow: /tagDisallow: /category/uncategorized# disallow all files ending with these extensionsDisallow: /.php$Disallow: /.js$Disallow: /.inc$Disallow: /.css$Disallow: /.txt$# disallow all files in /wp- directorysDisallow: /wp-/# disallow all files with? in urlDisallow: /*?

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)