Weighted Tensor Approximation Based on Apache Spark

• Main Authors: Qin Ding, 丁琴
• Other Authors: Yu-Ting Tsai
• Format: Others
• Language: en_US
• Published: 2017
• Online Access: http://ndltd.ncl.edu.tw/handle/2u2b69

碩士 === 元智大學 === 資訊工程學系 === 106 === This paper is aimed at implementing weighted tensor approximation (WTA) on Apache Spark, and achieving much less processing time than previous work. Unlike traditional tensor approximation (TA), this paper takes the validity of the input data into consideration during the data compression process. By giving different weights to valid and invalid data, we can eliminate the impact of invalid data and obtain better approximation results. However, because of the large amount of raw data, the compression process of WTA using just a stand-alone workstation not only takes a long time, but also poses a great challenge for hardware. In this paper, we investigate on how to parallelize the compression process for less processing time. We choose to implement WTA on Spark, whose computational performance is often faster than other distributed computing platforms, such as Apache Hadoop. The feasibility of WTA on Spark is achieved by transforming the original multilinear problem into a common linear one. The input tensor is also partitioned into small blocks to further reduce compression time.