2. server business whether to shield the spider crawling. Don’t think this is not possible, the original host house, mistakenly took love Shanghai spiders as hackers to shield, do not know how many people are in K station. If the host is shielding the spider crawling as soon as possible to change the host, time can not afford.
encountered such a problem, only two kinds: one, you shield the spider crawling. Two, you are a spider shield
1. check your domain name before the master, look at the domain name ever made illegal site. If it is not the spider climb is fully justified, like Maas before the forum before the domain name is Liuhecai station. Sogou and Google has never been a spider.
if it is found that the crawl log your site did not appear spiders visit, then we should consider their own site if the above situation, should be timely cleaning. And then submitted to the relevant search engine, or the use of the high quality of the chain and guide the spider crawling. (source: Mascherano – Micro Internet 贵族宝贝ma4.net reproduced indicate the source)
we shield the spider, the spider would not be able to grab our site pull.
two, you are shielding
just Shanghai dragon why saw a very interesting question, questions are as follows: (solving) why do I stand, never to love spiders in Shanghai. From the above problems can be seen, the owners encounter such problems must be very awful. Well to do their own site, did not think the spider isn’t to climb. So what is the reason for spiders not to crawl. I have to analyze.
1. first we want to detect your site robots.txt file, see the document writing is correct, whether there is a case of chaos, whether early station has shielding all the spider crawling instruction forget modified. What robots.txt detection normally directly to the webmaster tools to check on it
2. check yourself with the server site, whether there are serious violations of cheating, if any, the search engine is likely to shield the whole IP site, for the healthy development of their own website or independent IP server.
, your shield spider