en English is Íslenska

Article National and University Library of Iceland > Rit starfsmanna Lbs-Hbs >

Please use this identifier to cite or link to this item: http://hdl.handle.net/1946/6073

  • Incremental Crawling with Heritrix
  • September 2005
  • The Heritrix web crawler aims to be the world's first open source, extensible, web-scale, archival-quality web crawler. It has however been limited in its crawling strategies to snapshot crawling. This paper reports on work to add the ability to conduct incremental crawls to its capabilities. We first discuss the concept of incremental crawling as opposed to snapshot crawling and then the possible ways to design an effective incremental strategy. An overview is given of the implementation that we did, its limits and strengths are discussed. We then report on the results of initial experimentation with the new software which have gone well. Finally, we discuss issues that remain unresolved and possible future improvements.

  • Aug 27, 2010
  • http://hdl.handle.net/1946/6073

Files in This Item:
Filename Size VisibilityDescriptionFormat 
iwaw05-sigurdsson.pdf166.15 kBOpenHeildartextiPDFView/Open