Evaluating and improving the performances of job execution and data transmission in data grids, YouTube, and Hadoop YARN

• Main Authors: Lee, Ming-Chang, 李明昌
• Other Authors: Chen, Ying-ping
• Format: Others
• Language: en_US
• Published: 2015
• Online Access: http://ndltd.ncl.edu.tw/handle/99808945472329041277

博士 === 國立交通大學 === 資訊科學與工程研究所 === 103 === Recently, several different types of distributed computation and storage systems, such as data grids, YouTube, and Hadoop YARN system, have been widely employed around the world to respectively resolve complex scientific computation and storage problems, enable people to share videos, and process large scale of data and applications. In the above systems, bandwidth consumption and job execution performance are very important two issues. In data grids, several data replication algorithms have been proposed to shorten file transmission time, improve data access performance, and reduce bandwidth consumption. But none of them considers data access patterns, i.e., users’ access behaviors, which causes that data grids has longer data transmission delays and higher bandwidth consumption. YouTube utilizes a distributed memory caching scheme named Memcached to cache videos, and employs the least-recently-used (LRU for short) cache replacement algorithm to evict videos when Memcached runs out of space. However, LRU might increase network overhead and video retrieval time. On the other hand, Hadoop YARN provides several scheduling policies and supports queue hierarchy, while the corresponding impacts on different types of applications that are executable on Hadoop YARN are unknown. In order to solve the aforementioned problems, in this dissertation, we propose a Popular File Replicate First algorithm (PFRF for short) considered user access behavior to improve job turnaround time, data availability, and bandwidth cost in data grids. Next, we propose two Pareto-based algorithms for YouTube to reduce video fault and shorten video-retrieval time so that the network overhead and video retrieval time in YouTube can be improved. Finally, we study how the scheduling-policy combinations (SPCs for short) supported by Hadoop YARN with several different queue structures impact the performance of various types of applications.