Google wont read my robots.txt on s3
amazon-s3, googlebot, robots.txt, web-crawler
Solution
Which URL did you give Google for your bucket? You need to use the DNS-style `{bucket}.s3.amazonaws.com`, instead of the path-style `s3.amazonaws.com/{bucket}`.
Problem
As google is crawling our static content (stored on s3) we created a robots.txt in root directory (of the bucket).. as following ``` User-agent: * Disallow: / ``` Now since we wanted to remove exisiting content from google search .. we added the bucket in google webmaster a/c .. And requested for directory removal Now the issue is that google wont recognize or read the robots.txt (which is need to future blocking of google block).. We r getting this message in webmaster tools This site is not located at the top level for the domain. A robots.txt file is only valid when located in the highest-level directory and applies to all directories within the domain. The robots.txt file that applies to your site (if one exists) is located at http://s3.amazonaws.com/robots.txt. This page provides information on that file.