Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform

• Main Authors: Yi-Wei Huang黃奕崴, 黃奕崴
• Other Authors: Sheng-Tzong Cheng
• Format: Others
• Language: en_US
• Published: 2017
• Online Access: http://ndltd.ncl.edu.tw/handle/wmbc3v

碩士 === 國立成功大學 === 資訊工程學系 === 105 === Recent advances in sensor technology have led to the availability of a multitude of the sensor, e.g. sound, luminosity, and humidity. Huge raw data is a difficult problem to exploit and compute these data efficiently. Hadoop MapReduce has been used to solve this issue, but the operations which need iteration is not an efficient to handle these data. Hence, “In-memory Computing concept (IMC)” is come up to resolve the problem of Hadoop I/O bottleneck. In in-memory computing, the data is computed parallel in random access memory (RAM) instead of slow disk drives. We can train patterns and analyze large data frequently by IMC technique. However, IMC platform does not provide an effective reduce transmission scheme in the real-time system. It may limit some applications like wireless sensor network. It may be impractical for transmitting entire data from each sensor node, due to weak resource such as CPU, Memory, Power, etc. Compress data before sending is an effective way to make good use of sensor nodes limited power supply and make better the life of sensors. According to our observation, most of the sensor data has a similar pattern due to time dependence and spatial dependence. Therefore, we can improve compression efficiency by these characteristics. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform “Spark Streaming” which is used to collect data in real-time. We describe the whole system design and implement that provides a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with a little delay time in the soft real-time system.