Critical Task Scheduling for Data Stream Computing on Heterogeneous Clouds

• Main Authors: KUO, YEN-HSUAN, 郭晏瑄
• Other Authors: LAI, KUAN-CHOU
• Format: Others
• Language: zh-TW
• Published: 2017
• Online Access: http://ndltd.ncl.edu.tw/handle/p7fd7m

碩士 === 國立臺中教育大學 === 資訊工程學系 === 105 === Recently, Internet of Things is widely used in many different fields, such as logistics, medical care, science and urban development. Sensor data of these fields could be generated and collected through sensors. In the meantime, sensors created large and continuous data stream, which is called Big Data. In the application of Internet of things, the data collected through the sensor is continuously generated. This way of generating data is called Data Stream. Data stream processing usually requires a lot of calculation and immediate response, so how effective and immediate processing of large-scale streaming data is a new research topic. This study selects Apache Spark system to process data stream, this system is the use of In-Memory Computing calculation approach, eliminating the need for data access I/O time to speed up the data processing time. However, the scale of big data is too large, there is a lot of processing resources required in terms of calculation or storage. Cloud computing technology could be support enough computing capability. In Spark system, the stage execution order use FIFO algorithm to sort, the stage submit order is used as execution order. But the method not concern that the stage order change was affected the execution performance. In the worker node of the heterogeneous resources(offer), using Random approach to sort offer, but the offer of poor performance may be given priority to run job resulting in the impact of execution performance. So, this study proposes a Proactive Task Scheduling on Heterogeneous Resources (PTSHR) approach about these problem, the dependence of stages is generated by the profiling system, reorder the stages order and concern heterogeneous resources of the offers to sort, let better performance offer have priority to run task and tried to improve overall processing time.