The principle of
search engine is a spider crawler program, automatically collect the webpage on Internet and access to relevant information. In view of network security and privacy considerations, each site will set their own Robots protocol, to express the search engine, which content is willing to allow to be indexed by search engines, which are not allowed. The search engine will be in accordance with the Robots agreement to give yourself permission to crawl. The Robots protocol has become an international practice all must comply with the search engine. This is just like a normal person to go to other people, to knock on the door, get permission to enter the living room. Unless further license and invitation of the owner, or you are not allowed to enter the chamber, or in someone’s home, wandering.
to be accused of violating the robots agreement, due to the facts, Zhou Hongyi has not denied, but he also refuted said love Shanghai in the robots agreement to ban 360 crawler, unfair competition. Hong Bo said this, the robots protocol gives the site to prohibit any search crawler rights, the unfair competition with no relationship. 360 ignore the industry default rules, it is really unfair competition.
therefore, when the 360 comprehensive search two weeks Qianxin line blind Robots protocol, direct capture of unauthorized information data, it has been the industry questioned.
for the Qihoo 360 recently broke the disregard of international comprehensive search Roberts protocol, grab love Shanghai, noble baby and other search engines, resulting in many websites for security and privacy considerations do not allow the intranet information search engine grab was leaked issue, senior Internet observers Hong Bo pointed out that the search should abide by the search industry recognized game the rules, ignoring the rules, violation of rules is unfair competition, this behavior is not from law and government regulation to stop in time, will lead to industry chaos.
it is understood that love Shanghai website Robots agreement does not authorize the 360 search crawl, but ignore the 360 search settings, without the implementation of grasping behavior. Taking into account the many content source site banned search engines crawl pages mostly involved are stored in the server database, user privacy, passwords and other information. This means that 360 ignore the content source site robots.txt protocol set, will cause the stored in the server should not be searched the privacy information is searched, even directly in the search results displayed.
to do the search should abide by the search industry accepted the rules of the game, ignoring the rules, wanton violation of rules is really unfair competition." In Hong Bo’s opinion, love Shanghai does not prohibit all the crawler, and know the contents of Post Bar quiz, there is no love Shanghai banned the rules and the potential security risks of reptiles, this is the protection of the market order, protect the reasonable measure of user privacy. He pointed out that in 2008, Taobao was also.