Robot.txt file ! , How to use it ?

When optimizing your web site most webmasters don't consider using the robot.txt file. This is a very important file for your site. It lets the spiders and crawlers know what they can and cannot index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder.

Here is a list of variables that you can include in a robot.txt file and there meaning:

User-agent: In this field you can specify a specific robot to describe access policy for or a ?*? for all robots more explained in example.

Disallow: In the field you specify the files and folders not to include in the crawl.

Note: The # is to represent comments

Here are some examples of a robot.txt file .....

User-agent: *
Disallow:

The above would let all spiders index all content.

Here another

User-agent: *
Disallow: /cgi-bin/

The above would block all spiders from indexing the cgi-bin directory.

User-agent: googlebot
Disallow:

User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/

In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.

60 Websites to promote your articles on

Writing a good article is not enough. Sometimes, it needs a little help to spread around the Internet. Submiting your article to different websites brings you new visitors and backlinks too, which is good for SEO. Though you submit your articles to various article sites, but its really important to promote them in social sites. So here are 60 Websites to promote your articles on...

Courtesy area1.info