Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing
Efficient utilization of resources plays an important role in the performance of large scale task processing. In cases where heterogeneous types of resources are used within the same application, it is hard to achieve good utilization of all of the different types of resources. By taking advantage o...
Main Authors: | , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2018-07-01
|
Series: | Big Data and Cognitive Computing |
Subjects: | |
Online Access: | http://www.mdpi.com/2504-2289/2/3/15 |
id |
doaj-3f126dc33bd5475ba65d934c37677ee5 |
---|---|
record_format |
Article |
spelling |
doaj-3f126dc33bd5475ba65d934c37677ee52020-11-24T21:49:47ZengMDPI AGBig Data and Cognitive Computing2504-22892018-07-01231510.3390/bdcc2030015bdcc2030015Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data ProcessingMaarten Kollenstart0Edwin Harmsma1Erik Langius2Vasilios Andrikopoulos3Alexander Lazovik4Monitoring and Control Systems, TNO Groningen, Eemsgolaan 3, 9727 DW Groningen, The NetherlandsMonitoring and Control Systems, TNO Groningen, Eemsgolaan 3, 9727 DW Groningen, The NetherlandsMonitoring and Control Systems, TNO Groningen, Eemsgolaan 3, 9727 DW Groningen, The NetherlandsFaculty of Science and Engineering, University of Groningen, Nijenborgh 9, 9747 AG Groningen, The NetherlandsFaculty of Science and Engineering, University of Groningen, Nijenborgh 9, 9747 AG Groningen, The NetherlandsEfficient utilization of resources plays an important role in the performance of large scale task processing. In cases where heterogeneous types of resources are used within the same application, it is hard to achieve good utilization of all of the different types of resources. By taking advantage of recent developments in cloud infrastructure that enable the use of dynamic clusters of resources, and by dynamically altering the size of the available resources for all the different resource types, the overall utilization of resources, however, can be improved. Starting from this premise, this paper discusses a solution that aims to provide a generic algorithm to estimate the desired ratios of instance processing tasks as well as ratios of the resources that are used by these instances, without the necessity for trial runs or a priori knowledge of the execution steps. These ratios are then used as part of an adaptive system that is able to reconfigure itself to maximize utilization. To verify the solution, a reference framework which adaptively manages clusters of functionally different VMs to host a calculation scenario is implemented. Experiments are conducted based on a compute-heavy use case in which the probability of underground pipeline failures is determined based on the settlement of soils. These experiments show that the solution is capable of eliminating large amounts of under-utilization, resulting in increased throughput and lower lead times.http://www.mdpi.com/2504-2289/2/3/15cloud computingbig data processing and analyticsheterogeneous cloud resourcesindustrial case study |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Maarten Kollenstart Edwin Harmsma Erik Langius Vasilios Andrikopoulos Alexander Lazovik |
spellingShingle |
Maarten Kollenstart Edwin Harmsma Erik Langius Vasilios Andrikopoulos Alexander Lazovik Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing Big Data and Cognitive Computing cloud computing big data processing and analytics heterogeneous cloud resources industrial case study |
author_facet |
Maarten Kollenstart Edwin Harmsma Erik Langius Vasilios Andrikopoulos Alexander Lazovik |
author_sort |
Maarten Kollenstart |
title |
Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing |
title_short |
Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing |
title_full |
Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing |
title_fullStr |
Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing |
title_full_unstemmed |
Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing |
title_sort |
adaptive provisioning of heterogeneous cloud resources for big data processing |
publisher |
MDPI AG |
series |
Big Data and Cognitive Computing |
issn |
2504-2289 |
publishDate |
2018-07-01 |
description |
Efficient utilization of resources plays an important role in the performance of large scale task processing. In cases where heterogeneous types of resources are used within the same application, it is hard to achieve good utilization of all of the different types of resources. By taking advantage of recent developments in cloud infrastructure that enable the use of dynamic clusters of resources, and by dynamically altering the size of the available resources for all the different resource types, the overall utilization of resources, however, can be improved. Starting from this premise, this paper discusses a solution that aims to provide a generic algorithm to estimate the desired ratios of instance processing tasks as well as ratios of the resources that are used by these instances, without the necessity for trial runs or a priori knowledge of the execution steps. These ratios are then used as part of an adaptive system that is able to reconfigure itself to maximize utilization. To verify the solution, a reference framework which adaptively manages clusters of functionally different VMs to host a calculation scenario is implemented. Experiments are conducted based on a compute-heavy use case in which the probability of underground pipeline failures is determined based on the settlement of soils. These experiments show that the solution is capable of eliminating large amounts of under-utilization, resulting in increased throughput and lower lead times. |
topic |
cloud computing big data processing and analytics heterogeneous cloud resources industrial case study |
url |
http://www.mdpi.com/2504-2289/2/3/15 |
work_keys_str_mv |
AT maartenkollenstart adaptiveprovisioningofheterogeneouscloudresourcesforbigdataprocessing AT edwinharmsma adaptiveprovisioningofheterogeneouscloudresourcesforbigdataprocessing AT eriklangius adaptiveprovisioningofheterogeneouscloudresourcesforbigdataprocessing AT vasiliosandrikopoulos adaptiveprovisioningofheterogeneouscloudresourcesforbigdataprocessing AT alexanderlazovik adaptiveprovisioningofheterogeneouscloudresourcesforbigdataprocessing |
_version_ |
1725887465193996288 |