CryoEM Data Reduction and Compression
Behind every breakthrough dataset lies a growing challenge: the enormous storage demands of keeping raw and processed data accessible. Ada Lovelace Centre (ALC) has delivered a breakthrough in managing the vast amounts of data generated by cryogenic Electron Microscopy (cryoEM), a critical technique in structural biology.
The Challenge
CryoEM generates movies of biomolecules frozen at cryogenic temperatures. These are later converted into single-frame micrographs, but the original multi-frame movies are typically retained, resulting in terabytes of largely uncompressed data. Without intervention, this creates a strain on storage infrastructure, particularly at high-throughput facilities like Diamond Light Source’s Electron Bio-Imaging Centre (eBIC), which supports over 1,000 users annually.
The need for efficient data storage has never been greater. Advances in detector technology are driving a rapid increase in cryo-EM data volumes, with the Quantum C100 detector, developed through a collaboration between STFC, Quantum Detectors, the Rosalind Franklin Institute and the Medical Research Council Laboratory of Molecular Biology, representing a new generation of instrumentation capable of capturing more than 2,000 frames per second. As increasingly powerful detectors generate ever-larger datasets, the challenges of storing, transferring and archiving this data are becoming increasingly urgent.
"As cryo-EM datasets continue to grow at an unprecedented rate, efficient data management is becoming as important as advances in microscopes and detectors. Combining workflow-aware archiving with modern compression methods can substantially reduce storage demands, helping facilities manage rising infrastructure and energy costs while ensuring long-term sustainability of cryo-EM research”
Agnel, Praveen Joseph, Computational Scientist, Scientific Computing
Our Approach
The ALC team (graduates Mariam Demir and Willow Sparks, supported by computational scientist Agnel Praveen Joseph), assessed where complete data retention could be avoided and advanced compression techniques could be used without compromising scientific fidelity.
- Enhanced Motion Correction Workflow: By using RELION’s MotionCorr method to reuse already calculated alignment data, eBIC can now skip archiving the corrected micrographs— saving 48% of space in the archives. This will bring an enormous saving of around 5 petabytes per year (five million gigabytes)
- New Compression Framework: A user-friendly software library now enables the use of multiple compression methods without compromising the data. This includes BLOSC-Zstd, a high-performance algorithm that creates storage savings of 10-15% for raw movies stored as compressed TIFF image files.
The ALC software supports widely-used cryoEM image formats such as TIFF and MRC, integrates easily into automated workflows, and offers archiving abilities to streamline batch processing with metadata embedded. Through this, the team ensured usability for both research facilities and individual scientists.
Benefits
With the expectation of vastly reduced storage costs, improved archival reliability, and greater sustainability, this project will advance scientific techniques and bring benefits for multiple research communities
These advancements are not just local optimisations—CCP-EM and eBIC plan to share these developments with international databanks like EMDB (Electron Microscopy Data Bank) and EMPIAR (Electron Microscopy Public Image Archive), extending access to the wider structural biology community.
"eBIC generates around 1PB of data from each Krios microscope in a year, and the long-term storage of data is becoming an increasing problem as volumes get larger through developments in acquisition and processing. Little work has been done previously to develop methods to remove or compress data in ways that can be quickly regenerated if needed, so this project is providing significant benefits to eBIC and the wider cryo-EM community."
Stephen Riggs, Data Analysis Scientist, Diamond Light Source
Technical Insight
The compression framework tested eight algorithms, focusing on balancing space efficiency, speed, and data integrity. BLOSC-Zstd emerged as the best performer for raw movies, offering near-lossless fidelity with faster I/O performance suitable for hot storage as well. Support for HDF5 significantly improved throughput in handling large batch datasets by enabling chunked I/O operations and inline compression, positioning the toolset for integration with high-performance computing pipelines. Importantly, compatibility updates in RELION MotionCorr eliminate redundant micrographs, demonstrating that workflow-level changes can sometimes yield exponential gains in storage efficiency, beyond pure algorithmic compression.
This project exemplifies how ALC is reducing operational and environmental barriers in data-driven research—allowing scientists to focus on discovery rather than data storage challenges. Building on this work, a follow-up ALC project is evaluating new storage strategies and formats that have the potential to generate significant additional storage savings across the full workflow, while also enabling more efficient data access and faster data transfer.
If you want to find out more, you can get in touch here.
The Ada Lovelace Centre is part of STFC Scientific Computing and is funded by UKRI.
