A Load-balanced MapReduce Algorithm for Blocking-based Entity-resolution with Multiple Keys

• Main Authors: Yi-Chun Chiu, 邱羿郡
• Other Authors: Ming-Yen Lin
• Format: Others
• Language: en_US
• Published: 2013
• Online Access: http://ndltd.ncl.edu.tw/handle/08538221817158453241

碩士 === 逢甲大學 === 資訊工程學系 === 101 === Entity resolution (ER) is a long-lasting challenge in database management research. To detect records referring to the same entity across different data sources, an ER solution has to perform similarity computations for all pairs of the entities in the dataset. Most studies on blocking-based ER assume that one blocking key is associated with an entity. By distributing entities of the same blocking key, the number of comparisons in similarity computations can be reduced. An entity in reality may have multiple blocking keys in some applications. When the entities have a number of blocking keys, ER can be more efficient since two entities can form a similar pair only if they share several common keys. With the rapid growth of the data size, it becomes necessary to devise a blocking-based ER algorithm using the MapReduce framework to deal with the sheer volume of data collections today. Therefore, in this thesis, we propose a MapReduce algorithm to solve the ER problem for a huge collection of entities with multiple keys. The algorithm is characterized in the combination based blocking and the load-balanced matching. The combination based blocking utilizes the multiple keys to filter out unnecessary entity pairs for future matching. The load-balanced matching evenly distributes the required similarity computations to all the reducers in the matching step. The load-balanced design is to remove the bottleneck of skewed matching computations for a single node in a MapReduce framework. Our experiments using 1.4 million publication records show that the proposed algorithm is efficient and scalable.