zbMATH — the first resource for mathematics

Crawlets: Agents for high performance web search engines. (English) Zbl 1052.68954
Picco, Gian Pietro (ed.), Mobile agents. 5th international conference, MA 2001, Atlanta, GA, USA, December 2–4, 2001. Proceedings. Berlin: Springer (ISBN 3-540-42952-2). Lect. Notes Comput. Sci. 2240, 119-134 (2001).
Summary: Some of the reasons for unsatisfactory performance of today’s search engines are their centralized approach to web crawling and lack of explicit support from web servers. We propose a modification to conventional crawling in which a search engine uploads simple agents, called crawlets, to web sites. A crawlet crawls pages at a site locally and sends a compact summary back to the search engine. This not only reduces bandwidth requirements and network latencies, but also parallelizes crawling. Crawlets also provide an effective means for achieving the performance gains of personalized web servers, and can make up for the lack of cooperation from conventional web servers. The specialized nature of crawlets allows simple solutions to security and resource control problems, and reduces software requirements at participating web sites. In fact, we propose an implementation that requires no changes to web servers, but only the installation of a few (active) web pages at host sites.
For the entire collection see [Zbl 0979.68612].
68U99 Computing methodologies and applications
68T01 General topics in artificial intelligence
68M14 Distributed systems
Full Text: Link