Link Extraction for Crawling Flash on the Web

• Main Author: Antelius, Daniel
• Format: Others
• Language: English
• Published: Linköpings universitet, Institutionen för datavetenskap 2015
• Subjects: Flash; crawling; spidering; deep web; hidden web; virtual machine; interpretation; Computer Sciences; Datavetenskap (datalogi)
• Online Access: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-117604

The set of web pages not reachable using conventional web search engines is usually called the hidden or deep web. One client-side hurdle for crawling the hidden web is Flash files. This thesis presents a tool for extracting links from Flash files up to version 8 to enable web crawling. The files are both parsed and selectively interpreted to extract links. The purpose of the interpretation is to simulate the normal execution of Flash in the Flash runtime of a web browser. The interpretation is a low level approach that allows the extraction to occur offline and without involving automation of web browsers. A virtual machine is implemented and a set of limitations is chosen to reduce development time and maximize the coverage of interpreted byte code. Out of a test set of about 3500 randomly sampled Flash files the link extractor found links in 34% of the files. The resulting estimated web search engine coverage improvement is almost 10%.