Robots Text File Required says Google

Are you keeping your site or pages from being indexed?

Google’s Eric Kuan stated, in an online discussion that Google won’t index a website if they can’t access the robots.txt file of that site.

He said, “When Google encounters trouble crawling your robots.txt file, it will stop crawling the rest of your site to prevent it from crawling pages that have been blocked by the robots.txt file.”

If you are using WordPress, it includes a simple robots.txt file that’s dynamically generated from within the WP application.

But what happens if you want to change it to keep certain pages or even folders from being indexed? Enter the WP Robots Txt Plugin, it allows you to do that as needed. Here is a link to the plugin (copy / paste).

http://wordpress.org/plugins/wp-robots-txt/

Late Breaking News:

Google webspam team actively monitoring your Gmail account

In a recent post on his site Charles Floate, a UK based Search Engine Optimisation and Search Marketing Consultant performed a study to check if Google is reading Gmail users mail for webspam.

Here is an excerpt of what Charles has to report.

“This promoted me to do my own study. I did a fake correspondence with myself and apparently another client, in which we did a black hat site. I said I’d been doing things like buying links and other monitoring based keywords I thought the NSA.. I mean Google Webspam team would be monitoring.

The fact that Google is using underhanded tactics to begin targeting individual people and not just websites is a testament to how the company has changed from the ‘Do no evil’ stance they once had. I don’t believe we can trust Google with our privacy, our security or even to have good morals and do the right thing.”

Read more (copy / paste).: http://godofseo.co/case-study/webspam-team-actively-monitoring-gmail/#ixzz2qT47kh3q