Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform
碩士 === 國立成功大學 === 資訊工程學系 === 105 === Recent advances in sensor technology have led to the availability of a multitude of the sensor, e.g. sound, luminosity, and humidity. Huge raw data is a difficult problem to exploit and compute these data efficiently. Hadoop MapReduce has been used to solve this...
Main Authors: | , |
---|---|
Other Authors: | |
Format: | Others |
Language: | en_US |
Published: |
2017
|
Online Access: | http://ndltd.ncl.edu.tw/handle/wmbc3v |
id |
ndltd-TW-105NCKU5392047 |
---|---|
record_format |
oai_dc |
spelling |
ndltd-TW-105NCKU53920472019-05-15T23:47:02Z http://ndltd.ncl.edu.tw/handle/wmbc3v Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform 大數據串流平台上降低感測資料傳輸的方法 Yi-Wei Huang黃奕崴 黃奕崴 碩士 國立成功大學 資訊工程學系 105 Recent advances in sensor technology have led to the availability of a multitude of the sensor, e.g. sound, luminosity, and humidity. Huge raw data is a difficult problem to exploit and compute these data efficiently. Hadoop MapReduce has been used to solve this issue, but the operations which need iteration is not an efficient to handle these data. Hence, “In-memory Computing concept (IMC)” is come up to resolve the problem of Hadoop I/O bottleneck. In in-memory computing, the data is computed parallel in random access memory (RAM) instead of slow disk drives. We can train patterns and analyze large data frequently by IMC technique. However, IMC platform does not provide an effective reduce transmission scheme in the real-time system. It may limit some applications like wireless sensor network. It may be impractical for transmitting entire data from each sensor node, due to weak resource such as CPU, Memory, Power, etc. Compress data before sending is an effective way to make good use of sensor nodes limited power supply and make better the life of sensors. According to our observation, most of the sensor data has a similar pattern due to time dependence and spatial dependence. Therefore, we can improve compression efficiency by these characteristics. This study presents an effective reduce transmission scheme on a distributed real-time IMC platform “Spark Streaming” which is used to collect data in real-time. We describe the whole system design and implement that provides a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with a little delay time in the soft real-time system. Sheng-Tzong Cheng 鄭憲宗 2017 學位論文 ; thesis 52 en_US |
collection |
NDLTD |
language |
en_US |
format |
Others
|
sources |
NDLTD |
description |
碩士 === 國立成功大學 === 資訊工程學系 === 105 === Recent advances in sensor technology have led to the availability of a multitude of the sensor, e.g. sound, luminosity, and humidity. Huge raw data is a difficult problem to exploit and compute these data efficiently. Hadoop MapReduce has been used to solve this issue, but the operations which need iteration is not an efficient to handle these data. Hence, “In-memory Computing concept (IMC)” is come up to resolve the problem of Hadoop I/O bottleneck.
In in-memory computing, the data is computed parallel in random access memory (RAM) instead of slow disk drives. We can train patterns and analyze large data frequently by IMC technique. However, IMC platform does not provide an effective reduce transmission scheme in the real-time system. It may limit some applications like wireless sensor network. It may be impractical for transmitting entire data from each sensor node, due to weak resource such as CPU, Memory, Power, etc. Compress data before sending is an effective way to make good use of sensor nodes limited power supply and make better the life of sensors. According to our observation, most of the sensor data has a similar pattern due to time dependence and spatial dependence. Therefore, we can improve compression efficiency by these characteristics.
This study presents an effective reduce transmission scheme on a distributed real-time IMC platform “Spark Streaming” which is used to collect data in real-time. We describe the whole system design and implement that provides a high compression ratio in a small batch data from the source. It is expected to reduce data transmission with a little delay time in the soft real-time system.
|
author2 |
Sheng-Tzong Cheng |
author_facet |
Sheng-Tzong Cheng Yi-Wei Huang黃奕崴 黃奕崴 |
author |
Yi-Wei Huang黃奕崴 黃奕崴 |
spellingShingle |
Yi-Wei Huang黃奕崴 黃奕崴 Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform |
author_sort |
Yi-Wei Huang黃奕崴 |
title |
Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform |
title_short |
Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform |
title_full |
Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform |
title_fullStr |
Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform |
title_full_unstemmed |
Reduction Scheme for Sensor-Data Transmission on a Big Data Streaming Platform |
title_sort |
reduction scheme for sensor-data transmission on a big data streaming platform |
publishDate |
2017 |
url |
http://ndltd.ncl.edu.tw/handle/wmbc3v |
work_keys_str_mv |
AT yiweihuanghuángyìwǎi reductionschemeforsensordatatransmissiononabigdatastreamingplatform AT huángyìwǎi reductionschemeforsensordatatransmissiononabigdatastreamingplatform AT yiweihuanghuángyìwǎi dàshùjùchuànliúpíngtáishàngjiàngdīgǎncèzīliàochuánshūdefāngfǎ AT huángyìwǎi dàshùjùchuànliúpíngtáishàngjiàngdīgǎncèzīliàochuánshūdefāngfǎ |
_version_ |
1719155339888164864 |