Accelerating Big Data Processing for Particle Physics Experiments

ATLAS, one of the largest particle physics experiments at CERN's Large Hadron Collider, generates immense amounts of data. Researchers at the University of Chicago developed a method using FABRIC that drastically reduces the time and bandwidth required for data analysis.

"Data that once took a week to transfer and process can take as little as five minutes with our approach," said Ilija Vukotic from the University of Chicago.

A new approach to data filtering

For the new approach, data filtering is performed at CERN through FAB nodes physically situated there. FAB (FABRIC Across Borders) expanded the FABRIC testbed by connecting the core North American infrastructure to nodes in Asia and Europe. Using FABRIC's advanced networking capabilities, the researchers created Kubernetes clusters for high-throughput data processing.

The researchers recently demonstrated data filtering and delivery at 200 Gbps in a collaborative project with IRIS-HEP, tackling anticipated scale challenges for the High Luminosity LHC era.

How FABRIC enables this research

FABRIC's flexible networking configuration and scalable infrastructure allow the team to deploy and manage cloud-native services to address the growing challenges of real-time data processing for high-energy physics applications. The testbed's international reach through FAB nodes at CERN provides the physical proximity needed to filter data at the source, dramatically reducing the volume of data that needs to traverse long-distance networks.