SIMHAR - Smart Distributed Web Crawler for the Hidden Web Using SIM+Hash and Redis Server

Developing a distributed web crawler obliges major engineering challenges, all of which are eventually associated to scale. To retain corpus of search engine and a reasonable state of freshness, the crawler must be distributed over multiple computers. In distributed crawling, crawling agents are giv...

Full description

Bibliographic Details

Published in:	IEEE Access
Main Authors:	Sawroop Kaur, G. Geetha
Format:	Article
Language:	English
Published:	IEEE 2020-01-01
Subjects:	Distributed crawlers duplication detection hidden web web crawler
Online Access:	https://ieeexplore.ieee.org/document/9123854/