ESnet Turns on 400 Gbps Circuits to Five DOE National Labs

Today’s critical scientific research is being conducted by collaborators at far-flung national laboratories who require high-speed, low-latency access to high performance computing facilities and specialized instruments. In 2023, ESnet supercharged the bandwidth for five DOE national laboratories and user facilities, unleashing 400 Gigabit per second capability for Argonne National Laboratory (ANL), Brookhaven National Laboratory (BNL), Oak Ridge National Laboratory (ORNL), and Pacific Northwest National Laboratory (PNNL) as well as the National Energy Research Scientific Computing Center (NERSC). 

“Enabling 400 Gbps is transformational – it opens up new ways of performing science integrated across multiple DOE facilities.”

—Debbie Bard, NERSC Science Engagement and Workflows Group Lead

With this boost in capacity, scientists can process, analyze, visualize, share, and store the enormous quantities of research data at speeds much faster than previously possible. For example, ORNL leads the Earth Systems Grid Federation project (ESGF2) to improve the discovery, access, and storage of data used for Earth System models and simulations and climate change research. The collaborative project, which includes ANL and Lawrence Livermore National Laboratory, is already taking advantage of ESnet to move the planet’s largest collection of Earth System model output data between DOE high-performance computing and data facilities. The integration of ESnet is dramatically enhancing data-intensive science, facilitating data sharing, and improving access to simulation and AI platforms.

And at the Environmental Molecular Sciences Laboratory, an Office of Science user facility located at PNNL, researchers have embarked on a project known as the Molecular Observation Network, a nationwide effort to understand the processes that govern what happens to carbon in soil. More carbon resides in Earth’s soil than in the atmosphere and vegetation combined, and its fate plays a huge role in our climate. It’s an incredibly active environment, with scientists collecting reams of data about soil processes. The upgraded network makes it possible to transfer hundreds of gigabytes of project data in just a few minutes, not the hours previously required. The near real-time, constant data feed makes the autonomous experimentation planned at EMSL easier to implement.

As one of ASCR’s three high performance computing facilities, NERSC will see an immediate benefit from the 400 Gbps upgrade. Increasingly, NERSC’s more than 9,000 users move data in and out of the data center in two main ways. The first is by transferring large datasets (simulated or from an experiment) between ASCR facilities as scientists take advantage of the specialized computing available at each site. The other is by moving data from DOE experimental and observational facilities for analysis. This analysis can be very time-sensitive, with scientists requiring quick turnarounds to inform running experiments.

“The ESnet6 upgrade is essential to maintaining scientific productivity as data is increasingly transferred between experiment and computing sites,” said Debbie Bard, group lead for Science Engagement and Workflows at NERSC. “Enabling 400 Gbps is transformational – it opens up new ways of performing science integrated across multiple DOE facilities.”