NanoPyx - super-fast bioimage analysis powered by adaptive machine learning
Technologies: CARE, NanoJ, NanoJ-eSRRF, NanoJ-SQUIRREL, NanoJ-SRRF, NanoJ-VirusMapper and NanoPyx
Preprint published in bioRxiv, August 2023
Publisher: Cold Spring Harbor Laboratory
To overcome the challenges posed by large and complex microscopy datasets, we have developed NanoPyx, an adaptive bioimage analysis framework designed for high-speed processing. At the core of NanoPyx is the Liquid Engine, an agent-based machine-learning system that predicts acceleration strategies for image analysis tasks. Unlike traditional single-algorithm methods, the Liquid Engine generates multiple CPU and GPU code variations using a meta-programming system, creating a competitive environment where different algorithms are benchmarked against each other to achieve optimal performance under the user”s computational environment. In initial experiments focusing on super-resolution analysis methods, the Liquid Engine demonstrated an over 10-fold computational speed improvement by accurately predicting the ideal scenarios to switch between algorithmic implementations. NanoPyx is accessible to users through a Python library, code-free Jupyter notebooks, and a napari plugin, making it suitable for individuals regardless of their coding proficiency. Furthermore, the optimisation principles embodied by the Liquid Engine have broader implications, extending their applicability to various high-performance computing fields.