This week research data management service Globus announced that the largest single file transfer in the history of the company occurred. A team of scientists at Argonne National Laboratory led the movement of 2.9 petabytes of data – part of a research project involving three of the largest cosmological simulations to date.
The data was stored on the Oak Ridge Summit supercomputer, which is currently the world’s fastests supercomputer. Globus was tasked with moving files from disk to tape, a typical migration for researchers. Globus is software-as-a-service for research data management, used by hundreds of research institutions and high-performance computing (HPC) facilities worldwide. The service enables secure, reliable file transfer, sharing, and data publication for managing data throughout the research lifecycle.
“Storage is in general a very large problem in our community — the Universe is just very big, so our work can often generate a lot of data,” says Katrin Heitmann, Argonne physicist and computational scientist and an Oak Ridge National Laboratory Leadership Computing Facility (OLCF) Early Science user. “Using Globus to easily move the data around between different storage solutions and institutions for analysis is essential.
“Due to its uniqueness, the data is very precious and the analysis will take time,” says Dr. Heitmann. “The first step after the simulations were finished was to make a backup copy of the data to HPSS, so we can move the data back and forth between disk and tape and thus carry out the analysis in steps. We use Globus for this work due to its speed, reliability, and ease of use.”
“With exascale imminent, AI on the rise, HPC systems proliferating, and research teams more distributed than ever, fast, secure, reliable data movement and management are now more important than ever,” says Ian Foster, Globus co-founder and director of Argonne’s Data Science and Learning Division. “We tend to take these functions for granted, and yet modern collaborative research would not be possible without them.”
If you enjoyed this article and want to receive more valuable industry content like this, click here to sign up for our digital newsletters!
Leave a Reply