Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON
For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and...
Main Authors: | , , , , , , , , , , , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
EDP Sciences
2020-01-01
|
Series: | EPJ Web of Conferences |
Online Access: | https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_11006.pdf |
id |
doaj-661b310b04ec405486490abd4502ffaf |
---|---|
record_format |
Article |
spelling |
doaj-661b310b04ec405486490abd4502ffaf2021-08-02T22:58:34ZengEDP SciencesEPJ Web of Conferences2100-014X2020-01-012451100610.1051/epjconf/202024511006epjconf_chep2020_11006Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENONLassnig Mario0Barisits Martin1Laycock Paul J2Serfon Cédric3Vaandering Eric W4Ellis Katy5Illingworth Robert A.6Garonne Vincent7White John8Clark James A.9Fronze Gabriele10Joshi Rohini11Johnson Ian12Bauermeister Boris13European Organization for Nuclear ResearchEuropean Organization for Nuclear ResearchBrookhaven National LaboratoryBrookhaven National LaboratoryFermi National Accelerator LaboratoryScience and Technology Facilities CouncilFermi National Accelerator LaboratoryUniversity of OsloNordic e-Infrastructure CollaborationGeorgia Institute of TechnologyIstituto Nazionale di Fisica NucleareSquare Kilometre Array OrganisationScience and Technology Facilities CouncilStockholm UniversityFor many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_11006.pdf |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Lassnig Mario Barisits Martin Laycock Paul J Serfon Cédric Vaandering Eric W Ellis Katy Illingworth Robert A. Garonne Vincent White John Clark James A. Fronze Gabriele Joshi Rohini Johnson Ian Bauermeister Boris |
spellingShingle |
Lassnig Mario Barisits Martin Laycock Paul J Serfon Cédric Vaandering Eric W Ellis Katy Illingworth Robert A. Garonne Vincent White John Clark James A. Fronze Gabriele Joshi Rohini Johnson Ian Bauermeister Boris Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON EPJ Web of Conferences |
author_facet |
Lassnig Mario Barisits Martin Laycock Paul J Serfon Cédric Vaandering Eric W Ellis Katy Illingworth Robert A. Garonne Vincent White John Clark James A. Fronze Gabriele Joshi Rohini Johnson Ian Bauermeister Boris |
author_sort |
Lassnig Mario |
title |
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON |
title_short |
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON |
title_full |
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON |
title_fullStr |
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON |
title_full_unstemmed |
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON |
title_sort |
rucio beyond atlas: experiences from belle ii, cms, dune, eiscat3d, ligo/virgo, ska, xenon |
publisher |
EDP Sciences |
series |
EPJ Web of Conferences |
issn |
2100-014X |
publishDate |
2020-01-01 |
description |
For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON. |
url |
https://www.epj-conferences.org/articles/epjconf/pdf/2020/21/epjconf_chep2020_11006.pdf |
work_keys_str_mv |
AT lassnigmario ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT barisitsmartin ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT laycockpaulj ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT serfoncedric ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT vaanderingericw ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT elliskaty ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT illingworthroberta ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT garonnevincent ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT whitejohn ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT clarkjamesa ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT fronzegabriele ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT joshirohini ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT johnsonian ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon AT bauermeisterboris ruciobeyondatlasexperiencesfrombelleiicmsduneeiscat3dligovirgoskaxenon |
_version_ |
1721225972390297600 |