Intelligent Resource Management for Large-scale Data Stream Processing
With the increasing trend of using cloud computing resources, the efficient utilization of these resources becomes more and more important. Working with data stream processing is a paradigm gaining in popularity, with tools such as Apache Spark Streaming or Kafka widely available, and companies are...
Main Author: | |
---|---|
Format: | Others |
Language: | English |
Published: |
Uppsala universitet, Institutionen för informationsteknologi
2019
|
Subjects: | |
Online Access: | http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-391927 |
id |
ndltd-UPSALLA1-oai-DiVA.org-uu-391927 |
---|---|
record_format |
oai_dc |
spelling |
ndltd-UPSALLA1-oai-DiVA.org-uu-3919272019-08-27T04:35:13ZIntelligent Resource Management for Large-scale Data Stream ProcessingengStein, OliverUppsala universitet, Institutionen för informationsteknologi2019Engineering and TechnologyTeknik och teknologierWith the increasing trend of using cloud computing resources, the efficient utilization of these resources becomes more and more important. Working with data stream processing is a paradigm gaining in popularity, with tools such as Apache Spark Streaming or Kafka widely available, and companies are shifting towards real-time monitoring of data such as sensor networks, financial data or anomaly detection. However, it is difficult for users to efficiently make use of cloud computing resources and studies show that a lot of energy and compute hardware is wasted. We propose an approach to optimizing resource usage in cloud computing environments designed for data stream processing frameworks, based on bin packing algorithms. Test results show that the resource usage is substantially improved as a result, with future improvements suggested to further increase this. The solution was implemented as an extension of the HarmonicIO data stream processing framework and evaluated through simulated workloads. Student thesisinfo:eu-repo/semantics/bachelorThesistexthttp://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-391927UPTEC IT, 1401-5749 ; 19007application/pdfinfo:eu-repo/semantics/openAccess |
collection |
NDLTD |
language |
English |
format |
Others
|
sources |
NDLTD |
topic |
Engineering and Technology Teknik och teknologier |
spellingShingle |
Engineering and Technology Teknik och teknologier Stein, Oliver Intelligent Resource Management for Large-scale Data Stream Processing |
description |
With the increasing trend of using cloud computing resources, the efficient utilization of these resources becomes more and more important. Working with data stream processing is a paradigm gaining in popularity, with tools such as Apache Spark Streaming or Kafka widely available, and companies are shifting towards real-time monitoring of data such as sensor networks, financial data or anomaly detection. However, it is difficult for users to efficiently make use of cloud computing resources and studies show that a lot of energy and compute hardware is wasted. We propose an approach to optimizing resource usage in cloud computing environments designed for data stream processing frameworks, based on bin packing algorithms. Test results show that the resource usage is substantially improved as a result, with future improvements suggested to further increase this. The solution was implemented as an extension of the HarmonicIO data stream processing framework and evaluated through simulated workloads. |
author |
Stein, Oliver |
author_facet |
Stein, Oliver |
author_sort |
Stein, Oliver |
title |
Intelligent Resource Management for Large-scale Data Stream Processing |
title_short |
Intelligent Resource Management for Large-scale Data Stream Processing |
title_full |
Intelligent Resource Management for Large-scale Data Stream Processing |
title_fullStr |
Intelligent Resource Management for Large-scale Data Stream Processing |
title_full_unstemmed |
Intelligent Resource Management for Large-scale Data Stream Processing |
title_sort |
intelligent resource management for large-scale data stream processing |
publisher |
Uppsala universitet, Institutionen för informationsteknologi |
publishDate |
2019 |
url |
http://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-391927 |
work_keys_str_mv |
AT steinoliver intelligentresourcemanagementforlargescaledatastreamprocessing |
_version_ |
1719238116064100352 |