BotSeer

BotSeer was a Web-based information system and search tool used for research on Web robots and trends in Robot Exclusion Protocol deployment and adherence. It was created and designed by Yang Sun,[1] Isaac G. Councill,[2] Ziming Zhuang[3] and C. Lee Giles. BotSeer was in operation from 2007 to 2010, approximately.

History

BotSeer served as a resource for studying the regulation and behavior of Web robots as well as information about the creation of effective robots.txt files and crawler implementations. It was publicly available on the World Wide Web at the College of Information Sciences and Technology at the Pennsylvania State University.

BotSeer provided services including robots.txt searching, robot bias analysis,[4][5] and robot-generated log analysis. The prototype of BotSeer also allowed users to search 6,000 documentation files and source codes from 18 open source crawler projects.

BotSeer had indexed and analyzed 2.2 million robots.txt files obtained from 13.2 million websites, as well as a large Web server log of real-world robot behavior and related analysis. BotSeer's goals were to assist researchers, webmasters, web crawler developers and others with web robots related research and information needs. However, some people received BotSeer negatively, arguing that it contradicted the purpose of the robots.txt convention.[6]

BotSeer had also had set up a honeypot[7] to test the ethics, performance and behavior of web crawlers.

References

  • "Webmasters May Shape Search Results". Newsvine. Associated Press. November 28, 2007. Retrieved 2011-12-11.

Notes

  1. ^ "Yang Sun". Archived from the original on 2014-01-04. Retrieved 2019-06-13.
  2. ^ Isaac G. Councill Archived May 17, 2014, at the Wayback Machine
  3. ^ Ziming Zhuang Archived December 28, 2007, at the Wayback Machine
  4. ^ Yang Sun, Z. Zhuang, I. Councill, C.L. Giles, Determining Bias to Search Engines from Robots.txt Archived 2015-04-02 at the Wayback Machine, Proceedings of IEEE/WIC/ACM International Conference on Web Intelligence (WI 2007), 149-155, 2007.
  5. ^ "Zoom Web Media Offers Affordable Web Design, Development and SEO Services". www.zoomwebmedia.com. Archived from the original on 2012-11-30.
  6. ^ BotSeer? - SEO Best Practices Search Engine Forums
  7. ^ Web Robot Behavior and Performance Test at the Wayback Machine (archived December 22, 2008) (instead of unrelated current site )

Content Disclaimer

Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.

  1. The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
  2. There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
  3. It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
  4. Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
  5. Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.