×

Optimization issues in web search engines. (English) Zbl 1118.90039

Resende, Mauricio G. C. (ed.) et al., Handbook of optimization in telecommunications. New York, NY: Springer (ISBN 0-387-30662-5/hbk). 981-1015 (2006).
Summary: Crawlers are deployed by a web search engine for collecting information from different web servers in order to maintain the currency of its data base of web pages. We present studies on the optimization of web search engines from different perspectives. We first investigate the number of crawlers to be used by a search engine so as to maximize the currency of the data base without putting an unnecessary load on the network. Both the static setting, where crawlers are always active, and the dynamic setting where, crawlers may be activated/deactivated as a function of the state of the system, are addressed. We then consider the optimal scheduling of the visits of these crawlers to the web pages assuming these pages are modified at different rates. Finally, we briefly discuss some other optimization issues of web search engines, including page ranking and system optimization.
For the entire collection see [Zbl 1100.90001].

MSC:

90B35 Deterministic scheduling theory in operations research
90B22 Queues and service in operations research
90C40 Markov and semi-Markov decision processes

Software:

UbiCrawler
PDFBibTeX XMLCite