From: Harish [MSFT] on
Open the site in IE -> View source -> Search for robots. If an <no html
index> attribute is there it will not be crawl able.