An Efficient Dynamic Load-Balancing Large Scale Graph-Processing System

• Main Authors: Ming-Chia Kuo, 郭明嘉
• Other Authors: Pangfeng Liu
• Format: Others
• Language: en_US
• Published: 2018
• Online Access: http://ndltd.ncl.edu.tw/handle/kpmh4n

碩士 === 國立臺灣大學 === 資訊工程學研究所 === 106 === Since the introduction of pregel by Google, several large-scale graphprocessing systems have been introduced. These systems are based on the bulk synchronous parallel model or other similar models and use various strategies to optimize system performance. For example, Mizan monitors the workload of each worker to determine whether the workload between the workers is balanced with respect to the execution time. If the workload is unbalanced among workers, Mizan migrates nodes from overloaded workers to under-loaded workers to balance the load among workers and minimize the total execution time. On the basis of Mizan’s migration plan, we implement a graph-processing system called GPSer with an efficient re-partitioning graph scheme. Our system uses statistical tools, e.g., coefficient of variation and correlation coefficient, to modify the migration plan and determine whether the workloads are balanced among all workers. Our system can accurately monitor current workloads and decide whether to migrate nodes among workers to balance the load. When imbalance arises, the workload of all workers can quickly converge to a balanced state, thereby enhancing the system performance. In experiment our system outperforms the state-of-the-art dynamic load-balancing graph processing-system, such as Mizan.