×

UbiCrawler

swMATH ID: 13197
Software Authors: P. Boldi, B. Codenotti, M. Santini, S. Vigna
Description: UbiCrawler: a scalable fully distributed web crawler. We report our experience in implementing UbiCrawler, a scalable distributed Web crawler, using the Java programming language. The main features of UbiCrawler are platform independence, linear scalability, graceful degradation in the presence of faults, a very effective assignment function (based on consistent hashing) for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitations of the Java APIs, which prompted the authors to partially reimplement them.
Homepage: http://dl.acm.org/citation.cfm?id=1045969
Related Software: WebGraph; SparseMatrix; Mercator; ILUPACK; DIMACS; SNAP; R; rJava; xml2; httr; foreach; doParallel; SoftwareX; Rcpp; RCrawler; CRAN; Anderson; PATRICIA; LUBM; GraphChi
Referenced in: 32 Publications
all top 5

Referenced by 85 Authors

3 Boldi, Paolo
3 Carpentieri, Bruno
3 Shen, Zhaoli
3 Vigna, Sebastiano
3 Wen, Chun
2 Ferragina, Paolo
2 Fischer, Johannes
2 Gleich, David F.
2 Grossi, Roberto
2 Gu, Xian-Ming
2 Huang, Ting-Zhu
2 Santini, Massimo
1 Andersen, Reid
1 Apostolico, Alberto
1 Arz, Julian
1 Azgomi, Mohammad Abdollahi
1 Belazzougui, Djamal
1 Bieniecki, Wojciech
1 Brandes, Ulrik
1 Broß, Jan
1 Cao, Jie
1 Çatalyürek, Ümit V.
1 Chellapilla, Kumar
1 Constantine, Paul G.
1 Crescenzi, Pierluigi
1 Dayar, Tugrul
1 Drovandi, Guido
1 Fuketa, Masao
1 Gagie, Travis
1 Gebremedhin, Assefaw Hadish
1 Gog, Simon
1 Grabowski, Szymon
1 Habib, Michel A.
1 Hamann, Michael
1 Hauck, Matthias
1 He, Jing
1 Huang, Guangyan
1 Huynh, The Dang
1 Jordan, Michael Irwin
1 Kanda, Shunsuke
1 Karande, Chinmay
1 Kaya, Kamer
1 Kolda, Tamara Gibson
1 Langguth, Johannes
1 Lanzi, Leonardo
1 Latapy, Matthieu
1 Laura, Luigi
1 Liu, Zhen
1 Madduri, Kamesh
1 Mania, Horia
1 Manzini, Giovanni
1 Marino, Andrea
1 Mathieu, Fabien
1 Morita, Kazuhiro
1 Nain, Philippe
1 Nasri, Mitra
1 Nitto, Igor
1 Noyan, Gökçe N.
1 Ottaviano, Giuseppe
1 Pagh, Rasmus
1 Pan, Xinghao
1 Papailiopoulos, Dimitris S.
1 Paradies, Marcus
1 Pinar, Ali
1 Plantenga, Todd
1 Posenato, Roberto
1 Rajamanickam, Sivasankaran
1 Ramchandran, Kannan
1 Recht, Benjamin
1 Rossi, Ryan Anthony
1 Santaroni, Federico
1 Seshadhri, Comandur
1 Shariati, Saeed
1 Shi, Yong
1 Slota, George M.
1 Strasser, Ben
1 Su, Meng
1 Tan, Xueyuan
1 Task, Christine
1 Uçar, Bora
1 Venturini, Rossano
1 Viennot, Laurent
1 Wagner, Dorothea
1 Zhang, Yanchun
1 Zhou, Yunkai

Referencing Publications by Year