Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics
Detecting near duplicates on the web is challenging due to its volume and variety. Most of the previous studies require the setting of input parameters, making it difficult for them to achieve robustness across various scenarios without careful tuning. Recently, a universal and parameter-free simila...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
Hindawi Limited
2016-01-01
|
Series: | Mathematical Problems in Engineering |
Online Access: | http://dx.doi.org/10.1155/2016/3919043 |
id |
doaj-429cb6f2cd404be690926df34a42dbc1 |
---|---|
record_format |
Article |
spelling |
doaj-429cb6f2cd404be690926df34a42dbc12020-11-24T23:16:13ZengHindawi LimitedMathematical Problems in Engineering1024-123X1563-51472016-01-01201610.1155/2016/39190433919043Effective and Fast Near Duplicate Detection via Signature-Based Compression MetricsXi Zhang0Yuntao Yao1Yingsheng Ji2Binxing Fang3Key Laboratory of Trustworthy Distributed Computing and Service, Ministry of Education, Beijing University of Posts and Telecommunications, Beijing, ChinaKey Laboratory of Trustworthy Distributed Computing and Service, Ministry of Education, Beijing University of Posts and Telecommunications, Beijing, ChinaDepartment of Computer Science and Technology, Tsinghua University, Beijing, ChinaKey Laboratory of Trustworthy Distributed Computing and Service, Ministry of Education, Beijing University of Posts and Telecommunications, Beijing, ChinaDetecting near duplicates on the web is challenging due to its volume and variety. Most of the previous studies require the setting of input parameters, making it difficult for them to achieve robustness across various scenarios without careful tuning. Recently, a universal and parameter-free similarity metric, the normalized compression distance or NCD, has been employed effectively in diverse applications. Nevertheless, there are problems preventing NCD from being applied to medium-to-large datasets as it lacks efficiency and tends to get skewed by large object size. To make this parameter-free method feasible on a large corpus of web documents, we propose a new method called SigNCD which measures NCD based on lightweight signatures instead of full documents, leading to improved efficiency and stability. We derive various lower bounds of NCD and propose pruning policies to further reduce computational complexity. We evaluate SigNCD on both English and Chinese datasets and show an increase in F1 score compared with the original NCD method and a significant reduction in runtime. Comparisons with other competitive methods also demonstrate the superiority of our method. Moreover, no parameter tuning is required in SigNCD, except a similarity threshold.http://dx.doi.org/10.1155/2016/3919043 |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Xi Zhang Yuntao Yao Yingsheng Ji Binxing Fang |
spellingShingle |
Xi Zhang Yuntao Yao Yingsheng Ji Binxing Fang Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics Mathematical Problems in Engineering |
author_facet |
Xi Zhang Yuntao Yao Yingsheng Ji Binxing Fang |
author_sort |
Xi Zhang |
title |
Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics |
title_short |
Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics |
title_full |
Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics |
title_fullStr |
Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics |
title_full_unstemmed |
Effective and Fast Near Duplicate Detection via Signature-Based Compression Metrics |
title_sort |
effective and fast near duplicate detection via signature-based compression metrics |
publisher |
Hindawi Limited |
series |
Mathematical Problems in Engineering |
issn |
1024-123X 1563-5147 |
publishDate |
2016-01-01 |
description |
Detecting near duplicates on the web is challenging due to its volume and variety. Most of the previous studies require the setting of input parameters, making it difficult for them to achieve robustness across various scenarios without careful tuning. Recently, a universal and parameter-free similarity metric, the normalized compression distance or NCD, has been employed effectively in diverse applications. Nevertheless, there are problems preventing NCD from being applied to medium-to-large datasets as it lacks efficiency and tends to get skewed by large object size. To make this parameter-free method feasible on a large corpus of web documents, we propose a new method called SigNCD which measures NCD based on lightweight signatures instead of full documents, leading to improved efficiency and stability. We derive various lower bounds of NCD and propose pruning policies to further reduce computational complexity. We evaluate SigNCD on both English and Chinese datasets and show an increase in F1 score compared with the original NCD method and a significant reduction in runtime. Comparisons with other competitive methods also demonstrate the superiority of our method. Moreover, no parameter tuning is required in SigNCD, except a similarity threshold. |
url |
http://dx.doi.org/10.1155/2016/3919043 |
work_keys_str_mv |
AT xizhang effectiveandfastnearduplicatedetectionviasignaturebasedcompressionmetrics AT yuntaoyao effectiveandfastnearduplicatedetectionviasignaturebasedcompressionmetrics AT yingshengji effectiveandfastnearduplicatedetectionviasignaturebasedcompressionmetrics AT binxingfang effectiveandfastnearduplicatedetectionviasignaturebasedcompressionmetrics |
_version_ |
1725588329177546752 |