Application-level Optimization of End-to-end Data Transfer Throughput
For large-scale distributed applications, effective use of available network throughput and optimization of data transfer speed is crucial for end-to-end application performance. Today, many regional and national optical networking initiatives such as LONI, ESnet and Teragrid provide high speed netw...
Main Author: | |
---|---|
Other Authors: | |
Format: | Others |
Language: | en |
Published: |
LSU
2010
|
Subjects: | |
Online Access: | http://etd.lsu.edu/docs/available/etd-11152010-105813/ |
id |
ndltd-LSU-oai-etd.lsu.edu-etd-11152010-105813 |
---|---|
record_format |
oai_dc |
spelling |
ndltd-LSU-oai-etd.lsu.edu-etd-11152010-1058132013-01-07T22:53:05Z Application-level Optimization of End-to-end Data Transfer Throughput Yildirim, Esma Computer Science For large-scale distributed applications, effective use of available network throughput and optimization of data transfer speed is crucial for end-to-end application performance. Today, many regional and national optical networking initiatives such as LONI, ESnet and Teragrid provide high speed network connectivity to their users. However, majority of the users fail to obtain even a fraction of the theoretical speeds promised by these networks due to issues such as sub-optimal protocol tuning, disk bottleneck on the sending and/or receiving ends, and processor limitations. This implies that having high speed networks in place is important but not sufficient for the improvement of end-to-end data transfer throughput. Being able to effectively use these high speed networks is becoming more and more important. Optimization of the underlying protocol parameters at the application layer (i.e. opening multiple parallel TCP streams, tuning the TCP buffer size and I/O block size) is one way of improving the network transfer throughput. On the other hand, end-to-end data transfer throughput bottleneck on high performance networking systems occur mostly at the participating storage systems rather than the network. The performance of a storage system heavily depends on the speed of its disk and CPU subsystems. Thus, it is critical to estimate the storage system's bandwidth at both endpoints in addition to the network bandwidth. Disk bottleneck can be eliminated by the use of multiple disks (data striping), and CPU bottleneck can be eliminated by the use of multiple processors (parallelism). In this dissertation, we develop application-level models to predict the best combination of protocol parameters for optimal network performance, including the number of parallel data streams, protocol buffer size; and integration of disk and CPU speed parameters into the performance model to predict the optimal number of disk and CPU striping for the best end-to-end data throughput. These models will be made available to the community for use in data transfer tools, schedulers, and high-level planners. Kosar, Tevfik Chen, Jianhua Park, Seung-Jong Allen, Gabrielle D. Busch, Konstantin Li, Guoqiang LSU 2010-11-16 text application/pdf http://etd.lsu.edu/docs/available/etd-11152010-105813/ http://etd.lsu.edu/docs/available/etd-11152010-105813/ en unrestricted I hereby certify that, if appropriate, I have obtained and attached herein a written permission statement from the owner(s) of each third party copyrighted matter to be included in my thesis, dissertation, or project report, allowing distribution as specified below. I certify that the version I submitted is the same as that approved by my advisory committee. I hereby grant to LSU or its agents the non-exclusive license to archive and make accessible, under the conditions specified below and in appropriate University policies, my thesis, dissertation, or project report in whole or in part in all forms of media, now or hereafter known. I retain all other ownership rights to the copyright of the thesis, dissertation or project report. I also retain the right to use in future works (such as articles or books) all or part of this thesis, dissertation, or project report. |
collection |
NDLTD |
language |
en |
format |
Others
|
sources |
NDLTD |
topic |
Computer Science |
spellingShingle |
Computer Science Yildirim, Esma Application-level Optimization of End-to-end Data Transfer Throughput |
description |
For large-scale distributed applications, effective use of available network throughput and optimization of data transfer speed is crucial for end-to-end application performance. Today, many regional and national optical networking initiatives such as LONI, ESnet and Teragrid provide high speed network connectivity to their users. However, majority of the users fail to obtain even a fraction of the theoretical speeds promised by these networks due to issues such as sub-optimal protocol tuning, disk bottleneck on the sending and/or receiving ends, and processor limitations. This implies that having high speed networks in place is important but not sufficient for the improvement of end-to-end data transfer throughput. Being able to effectively use these high speed networks is becoming more and more important.
Optimization of the underlying protocol parameters at the application layer (i.e. opening multiple parallel TCP streams, tuning the TCP buffer size and I/O block size) is one way of improving the network transfer throughput. On the other hand, end-to-end data transfer throughput bottleneck on high performance networking systems occur mostly at the participating storage systems rather than the network. The performance of a storage system heavily depends on the speed of its disk and CPU subsystems. Thus, it is critical to estimate the storage system's bandwidth at both endpoints in addition to the network bandwidth. Disk bottleneck can be eliminated by the use of multiple disks (data striping), and CPU bottleneck can be eliminated by the use of multiple processors (parallelism).
In this dissertation, we develop application-level models to predict the best combination of protocol parameters for optimal network performance, including the number of parallel data streams, protocol buffer size; and integration of disk and CPU speed parameters into the performance model to predict the optimal number of disk and CPU striping for the best end-to-end data throughput. These models will be made available to the community for use in data transfer tools, schedulers, and high-level planners. |
author2 |
Kosar, Tevfik |
author_facet |
Kosar, Tevfik Yildirim, Esma |
author |
Yildirim, Esma |
author_sort |
Yildirim, Esma |
title |
Application-level Optimization of End-to-end Data Transfer Throughput |
title_short |
Application-level Optimization of End-to-end Data Transfer Throughput |
title_full |
Application-level Optimization of End-to-end Data Transfer Throughput |
title_fullStr |
Application-level Optimization of End-to-end Data Transfer Throughput |
title_full_unstemmed |
Application-level Optimization of End-to-end Data Transfer Throughput |
title_sort |
application-level optimization of end-to-end data transfer throughput |
publisher |
LSU |
publishDate |
2010 |
url |
http://etd.lsu.edu/docs/available/etd-11152010-105813/ |
work_keys_str_mv |
AT yildirimesma applicationleveloptimizationofendtoenddatatransferthroughput |
_version_ |
1716477924599660544 |