Comparison of Three Vertical Search Spiders - Chau, Michael (IEEE, 2003)
Artificial Intelligence Lab, Department of MIS, University of Arizona === Spiders are the software agents that search engines use to collect content for their databases. We investigated algorithms to improve the performance of vertical search engine spiders. The investigation addressed three approaches: a breadth-first graph-traversal algorithm with no heuristics to refine the search process, a best-first traversal algorithm that used a hyperlink-analysis heuristic, and a spreading-activation algorithm based on modeling the Web as a neural network.