EJFAT Achieves Real-Time, Terabit-Scale Data Streaming
In April 2024, scientists at Thomas Jefferson National Accelerator Facility (Jefferson Lab) and ESnet achieved a major breakthrough for the transmission and processing of scientific data: they streamed raw physics data at 100 gigabits per second from Virginia to California, where it was processed on the Perlmutter supercomputer at NERSC and the results returned in real time — with no buffering, storage delays, or data loss.
The test used the ESnet–JLab FPGA Accelerated Transport (EJFAT) prototype, a hardware-accelerated load balancer designed to connect high-speed scientific instruments directly to remote DOE high-performance computing (HPC) facilities. EJFAT uses Field Programmable Gate Arrays (FPGAs) to shape and route data within nanoseconds, dynamically balancing workloads across thousands of compute cores.
In this demonstration, 40 Perlmutter nodes (over 10,000 cores) continuously processed replayed data from Jefferson Lab’s CLAS12 detector, sending results back every 400 microseconds — reducing turnaround from the 14 hours typical of traditional file-based workflows to near-instant feedback. Later tests replicated the 100 Gbps throughput and achieved simultaneous streaming across ~20,000 cores at NERSC, Oak Ridge Leadership Computing Facility, and the FABRIC testbed, in anticipation of projected demands of next-generation experiments like the Electron-Ion Collider.
EJFAT’s success marks the first production-level demonstration of fully remote, distributed data stream processing for real physics data across multiple DOE facilities — a critical step toward the Integrated Research Infrastructure (IRI) vision of seamlessly linking instruments, compute, and storage across the DOE complex. With its scalable, stateless design, EJFAT is quickly being adapted to other time-sensitive facilities — from light sources to fusion devices — enabling scientists to steer experiments in real time and accelerating discovery across disciplines.