Preserving the Chronological Versions of HDFS Files in Hadoop Clusters

• Main Authors: CHIEN,TING-YU, 簡霆毓
• Other Authors: YEH,TSOZEN
• Format: Others
• Language: zh-TW
• Published: 2016
• Online Access: http://ndltd.ncl.edu.tw/handle/06634888613504148822

碩士 === 輔仁大學 === 資訊工程學系碩士班 === 104 === Cloud platform in recent years has been widely used in various area, including Hadoop platform environment which is the most widely used. Initially Hadoop provides a simple, scalable, efficient cloud computing and cloud storage architecture. With the rapid advances in data analysis and large demand computing, the new development and application of the Hadoop ecosystem approach has been increasingly richer, its efficacy also improves. Hadoop distributed file system, HDFS(Hadoop Distributed File System) is the default Hadoop file system, HDFS structure mainly includes two roles, NameNode and DataNode. Basic HDFS will be configured as a cluster with one NameNode and host and multiple DataNode, NameNode keeps metadata to manage all archives namespace and location information of storage blocks, DataNode is responsible for storing files block, and their duplicates. During the early development of Hadoop, in order to meet the performance of using large files stored some of the basic features of the UNIX file system were not available. So Hadoop does not allow users to modify HDFS file content until append and truncate features appeared later. Meanwhile, Hadoop 2.2 officially added snapshot capabilities. It can specify the time to do a complete backup of the entire directory of files. Frequent snapshots many will have the similar backups. However, if there is a change made to the content between two snapshots, the changes will not be rewarded accordingly. In the past, a more the complete file version history can only be done through increasing the numbers of snapshots. However, often only a small number of files in the snapshots really require frequent backups. Even so, change between two consecutive snapshots are still not rewarded as described above. We developed new a mechanism by using existing HDFS append feature to save all versions of individual files, we also provide uses an easy way to retrieve those different file versions kept. As a result, were can we are design to easily identify the contents of different versions of individual files.