Build Your Own Arcade Controls Forum

Main => Forum/Website Discussion => Topic started by: krick on December 30, 2010, 10:23:06 pm

Title: Saint, hide the sandbox site with robots.txt
Post by: krick on December 30, 2010, 10:23:06 pm

Stuff from the forumsandbox site is showing up in Google.

You should hide the sandbox site with a custom robots.txt just for the sandbox...

http://www.robotstxt.org/faq/prevent.html (http://www.robotstxt.org/faq/prevent.html)

The sooner you get on this, the better since it's almost impossible to remove stuff from Google once it's been crawled.

Title: Re: Saint, hide the sandbox site with robots.txt
Post by: saint on December 30, 2010, 11:01:31 pm

Thanks - done.

Title: Re: Saint, hide the sandbox site with robots.txt
Post by: Crazy Cooter on December 31, 2010, 11:10:22 pm

It just showed up for me.  Seems to still have the links.

Title: Re: Saint, hide the sandbox site with robots.txt
Post by: krick on January 01, 2011, 02:03:30 am

Quote from: Crazy Cooter on December 31, 2010, 11:10:22 pm

It just showed up for me.  Seems to still have the links.

The robots.txt looks good to me...

http://forumsandbox.arcadecontrols.com/robots.txt (http://forumsandbox.arcadecontrols.com/robots.txt)