Automate Scientific Workflow Execution between Local Cluster and Cloud

• Main Authors: Hao Qian, Daniel Andresen
• Format: Article
• Language: English
• Published: Atlantis Press 2016-01-01
• Series: International Journal of Networked and Distributed Computing (IJNDC)
• Subjects: code offloading; scientific workflow; distributed computing; scheduling; cloud computing
• Online Access: https://www.atlantis-press.com/article/25846121.pdf

Scientific computational experiments often span multiple computational and analytical steps, and during execution, researchers need to store, access, transfer, and query information. Scientific workflow is a powerful tool to stream-line and organize scientific application. Numbers of tools have been developed to help build scientific workflows, they provide mechanisms for creating workflow but lack a native scheduling system for determining where code should be executed. This paper presents Emerald, a system that adds sophisticated computation offloading capabili-ties to scientific workflows. Emerald automatically offloads computation intensive steps of scientific workflow to the cloud in order to enhance workflow performance. Emerald minimizes the burden on developers to build work-flows with computation offloading ability by providing easy-to-use API. Evaluation showed that Emerald can ef-fectively reduce up to 55% of execution time for scientific applications.