Technology

CARE (Content-Aware image Restoration) is a deep learning based approach for restoring and enhancing fluorescence microscopy images. It uses convolutional neural networks to learn complex mappings between low quality input images and higher quality target images, allowing the restoration of images beyond the physical limits of the microscope.

CARE networks are trained on pairs of images depicting the same content - one image acquired under suboptimal conditions (low signal-to-noise ratio, low resolution etc.) serves as the input, while the other image shows the desired target quality. The type of restoration determines what constitutes the input and target images. For example, for image denoising the input could be a low laser power acquisition exhibiting high noise, while the target is a clean image obtained with higher laser power.

The pairs of input and target images showing the same content are used to train a neural network to perform the restoration mapping. While creating such training data is typically difficult for image processing tasks, the authors of CARE introduce several clever strategies to obtain suitable data without extensive manual labor:

1) For denoising, they interleave low and high laser power acquisitions during imaging to obtain registered image pairs of the same sample.

2) For isotropic resolution restoration, they use the higher resolved lateral planes of a volume as target and synthetically generate matching axial planes with lower resolution.

3) For super-resolution, they create simulated ground truth images of structures of interest and realistically degrade them to yield synthetic input images.

Using these strategies, thousands of aligned input-target pairs showing diverse biological image contents are generated. A convolutional neural network with a U-Net-like architecture is then trained to restore low quality inputs to match the target images. The particular network architecture and training process is tailored to each specific restoration type.

Once the CARE network is trained, it can be applied to new unseen microscopy images that were acquired under suboptimal conditions in order to computationally restore and enhance image quality beyond the physical limits of the microscope.

Some concrete examples demonstrated:

• Denoising time-lapse recordings of live samples at very high frame rates, since a CARE network can restore the individual noisy frames to a quality otherwise only achieved through longer exposure or higher laser power not compatible with living samples

• Projection of fluorescently labeled cell layers from noisy 3D volumes at high speed, by using a network trained to jointly perform projection and denoising

• Isotropic restoration of anisotropic volumes through a network that learns to computationally restore missing axial resolution from lateral planes

• Super-resolution enhancement of microtubules and secretory granules from widefield microscopy through a network trained only on simulated structures

CARE leverages recent advances in deep learning to computationally improve microscopy images beyond the physical limits of the respective microscope. It allows trading off imaging parameters that ordinarily cannot be optimized jointly, thereby enhancing observable biological phenomena. Through tailored training data generation and network design, CARE delivers marked improvements in image quality for a variety of restoration problems in fluorescence microscopy.

---

Publications featuring CARE

Harnessing artificial intelligence to reduce phototoxicity in live imaging Estibaliz Gómez-de-Mariscal, Mario Del Rosario, Joanna W. Pylvänäinen, Guillaume Jacquemet, Ricardo Henriques Perspective published in Journal of Cell Science, February 2024 Technologies: BioImage Model Zoo, CARE, DeepBacs, NanoJ-SRRF and ZeroCostDL4Mic Funded by: CZI, EMBO, ERC, H2021 and H2022 DOI: 10.1242/jcs.261545

High-fidelity 3D live-cell nanoscopy through data-driven enhanced super-resolution radial fluctuation Romain F. Laine, Hannah S. Heil, Simao Coelho, Jonathon Nixon-Abell, Angélique Jimenez, Theresa Wiesner, Damián Martínez, Tommaso Galgani, Louise Régnier, Aki Stubb, Gautier Follain, Samantha Webster, Jesse Goyette, Aurelien Dauphin, Audrey Salles, Siân Culley, Guillaume Jacquemet, Bassam Hajj, Christophe Leterrier, Ricardo Henriques Paper published in Nature Methods, November 2023 Technologies: CARE, NanoJ, NanoJ-eSRRF, NanoJ-SQUIRREL, NanoJ-SRRF, NanoPyx and Nuclear-Pores as references Funded by: CZI, EMBO, ERC, FCT, H2021, H2022, InnOValley and Wellcome Trust News: Photonics.com, The Science Times, Optics.org and Phys.org Blogs: Springer Nature Protocols and Methods Community DOI: 10.1038/s41592-023-02057-w

Transertion and cell geometry organize the Escherichia coli nucleoid during rapid growth Christoph Spahn, Stuart Middlemiss, Estibaliz Gómez-de-Mariscal, Ricardo Henriques, Helge B. Bode, Séamus Holden, Mike Heilemann Preprint published in bioRxiv, October 2023 Technologies: CARE, DeepBacs and ZeroCostDL4Mic Funded by: EMBO, ERC, H2021 and H2022 DOI: 10.1101/2023.10.16.562172

NanoPyx - super-fast bioimage analysis powered by adaptive machine learning Bruno M. Saraiva, Inês M. Cunha, António D. Brito, Gautier Follain, Raquel Portela, Robert Haase, Pedro M. Pereira, Guillaume Jacquemet, Ricardo Henriques Preprint published in bioRxiv, August 2023 Technologies: CARE, NanoJ, NanoJ-eSRRF, NanoJ-SQUIRREL, NanoJ-SRRF, NanoJ-VirusMapper and NanoPyx Funded by: CZI, EMBO, ERC, H2021 and H2022 DOI: 10.1101/2023.08.13.553080

CLEM-Reg - An automated point cloud based registration algorithm for correlative light and volume electron microscopy Daniel Krentzel, Matouš Elphick, Marie-Charlotte Domart, Christopher J. Peddie, Romain F. Laine, Ricardo Henriques, Lucy M. Collinson, Martin L. Jones Preprint published in bioRxiv, May 2023 Technologies: CARE and ZeroCostDL4Mic Funded by: CZI and ERC DOI: 10.1101/2023.05.11.540445

Live-cell imaging in the deep learning era Joanna W Pylvänäinen, Estibaliz Gómez-de-Mariscal, Ricardo Henriques, Guillaume Jacquemet Review published in Current Opinion in Cell Biology, January 2023 Technologies: BioImage Model Zoo, CARE, DeepBacs, Fast4DReg, NanoJ, NanoJ-eSRRF, NanoJ-Fluidics, NanoJ-SRRF and ZeroCostDL4Mic Funded by: CZI, EMBO, ERC and H2021 DOI: 10.1016/j.ceb.2023.102271

Fast4DReg–fast registration of 4D microscopy datasets Joanna W Pylvänäinen, Romain F Laine, Bruno MS Saraiva, Sujan Ghimire, Gautier Follain, Ricardo Henriques, Guillaume Jacquemet Paper published in Journal of Cell Science, January 2023 Technologies: CARE, Fast4DReg, NanoJ and ZeroCostDL4Mic Funded by: CZI and ERC DOI: 10.1242/jcs.260728

Roadmap on Deep Learning for Microscopy Giovanni Volpe, Carolina Wählby, Lei Tian, Michael Hecht, Artur Yakimovich, Kristina Monakhova, Laura Waller, Ivo F Sbalzarini, Christopher A Metzler, Mingyang Xie, Kevin Zhang, Isaac CD Lenton, Halina Rubinsztein-Dunlop, Daniel Brunner, Bijie Bai, Aydogan Ozcan, Daniel Midtvedt, Hao Wang, Nataša Sladoje, Joakim Lindblad, Jason T Smith, Marien Ochoa, Margarida Barroso, Xavier Intes, Tong Qiu, Li-Yu Yu, Sixian You, Yongtao Liu, Maxim A Ziatdinov, Sergei V Kalinin, Arlo Sheridan, Uri Manor, Elias Nehme, Ofri Goldenberg, Yoav Shechtman, Henrik K Moberg, Christoph Langhammer, Barbora Špačková, Saga Helgadottir, Benjamin Midtvedt, Aykut Argun, Tobias Thalheim, Frank Cichos, Stefano Bo, Lars Hubatsch, Jesus Pineda, Carlo Manzo, Harshith Bachimanchi, Erik Selander, Antoni Homs-Corbera, Martin Fränzl, Kevin de Haan, Yair Rivenson, Zofia Korczak, Caroline Beck Adiels, Mite Mijalkov, Dániel Veréb, Yu-Wei Chang, Joana B Pereira, Damian Matuszewski, Gustaf Kylberg, Ida-Maria Sintorn, Juan C Caicedo, Beth A Cimini, Muyinatu A Lediju Bell, Bruno M Saraiva, Guillaume Jacquemet, Ricardo Henriques, Wei Ouyang, Trang Le, Estibaliz Gómez-de-Mariscal, Daniel Sage, Arrate Muñoz-Barrutia, Ebba Josefson Lindqvist, Johanna Bergman Preprint published in arXiv, January 2023 Technologies: BioImage Model Zoo, CARE and ZeroCostDL4Mic Funded by: CZI, EMBO, ERC and H2021 DOI: 10.48550/arXiv.2303.03793

DeepBacs for multi-task bacterial image analysis using open-source deep learning approaches Christoph Spahn, Estibaliz Gómez-de-Mariscal, Romain F. Laine, Pedro M. Pereira, Lucas von Chamier, Mia Conduit, Mariana G. Pinho, Guillaume Jacquemet, Séamus Holden, Mike Heilemann, Ricardo Henriques Paper published in Communications Biology, July 2022 Technologies: BioImage Model Zoo, CARE, DeepBacs, NanoJ, NanoJ-SQUIRREL, NanoJ-SRRF and ZeroCostDL4Mic Funded by: CZI, ERC and Wellcome Trust DOI: 10.1038/s42003-022-03634-z

Expansion Microscopy on Saccharomyces cerevisiae AG Korovesi, L Morgado, M Fumasoni, R Henriques, HS Heil, M Del Rosario Paper published in microPublication Biology, January 2022 Technologies: CARE Funded by: EMBO, ERC and Wellcome Trust DOI: 10.17912/micropub.biology.000566

eHooke - A tool for automated image analysis of spherical bacteria based on cell cycle progression Bruno M. Saraiva, Ludwig Krippahl, Sérgio R. Filipe, Ricardo Henriques, Mariana G. Pinho Paper published in Biological Imaging, September 2021 Technologies: CARE Funded by: ERC and Wellcome Trust DOI: 10.1017/s2633903x21000027

FtsZ treadmilling is essential for Z-ring condensation and septal constriction initiation in Bacillus subtilis cell division Kevin D. Whitley, Calum Jukes, Nicholas Tregidgo, Eleni Karinou, Pedro Almada, Yann Cesbron, Ricardo Henriques, Cees Dekker, Séamus Holden Paper published in Nature Communications, April 2021 Technologies: CARE and DeepAutoFocus Funded by: BBSRC and Wellcome Trust News: Nanowerk DOI: 10.1038/s41467-021-22526-0

Democratising deep learning for microscopy with ZeroCostDL4Mic Lucas von Chamier, Romain F. Laine, Johanna Jukkala, Christoph Spahn, Daniel Krentzel, Elias Nehme, Martina Lerche, Sara Hernández-Pérez, Pieta K. Mattila, Eleni Karinou, Séamus Holden, Ahmet Can Solak, Alexander Krull, Tim-Oliver Buchholz, Martin L. Jones, Loïc A. Royer, Christophe Leterrier, Yoav Shechtman, Florian Jug, Mike Heilemann, Guillaume Jacquemet, Ricardo Henriques Paper published in Nature Communications, April 2021 Technologies: CARE, FBSR-TFM, NanoJ, NanoJ-SQUIRREL, NanoJ-SRRF and ZeroCostDL4Mic Funded by: EMBO, ERC and Wellcome Trust News: AZO Life Sciences, Drug Target Review, Nanotechnology Now and The Medical News Blogs: Microbiome Digest - Bik's Picks DOI: 10.1038/s41467-021-22518-0

Single-Molecule Super-Resolution Imaging of T-Cell Plasma Membrane CD4 Redistribution upon HIV-1 Binding Yue Yuan, Caron A. Jacobs, Isabel Llorente Garcia, Pedro M. Pereira, Scott P. Lawrence, Romain F. Laine, Mark Marsh, Ricardo Henriques Paper published in Viruses, January 2021 Technologies: CARE Funded by: ERC DOI: 10.3390/v13010142

Avoiding a replication crisis in deep-learning-based bioimage analysis Romain F Laine, Ignacio Arganda-Carreras, Ricardo Henriques, Guillaume Jacquemet Perspective published in Nature methods, January 2021 Technologies: CARE and ZeroCostDL4Mic Funded by: EMBO, ERC and Wellcome Trust DOI: 10.1038/s41592-021-01284-3

The field guide to 3D printing in microscopy Mario Del Rosario, Hannah S Heil, Afonso Mendes, Vittorio Saggiomo, Ricardo Henriques Review published in Adv. Biol., January 2021 Technologies: CARE, NanoJ, NanoJ-Fluidics and NanoJ-SRRF Funded by: EMBO, ERC and Wellcome Trust DOI: 10.1002/adbi.202100994

Closed mitosis requires local disassembly of the nuclear envelope Gautam Dey, Siân Culley, Scott Curran, Uwe Schmidt, Ricardo Henriques, Wanda Kukulski, Buzz Baum Paper published in Nature, August 2020 Technologies: CARE, NanoJ, NanoJ-SRRF and Nuclear-Pores as references Funded by: BBSRC and Wellcome Trust News: Nature Asia DOI: 10.1038/s41586-020-2648-3

The cell biologist's guide to super-resolution microscopy Guillaume Jacquemet, Alexandre F. Carisey, Hellyeh Hamidi, Ricardo Henriques, Christophe Leterrier Review published in Journal of Cell Science, June 2020 Technologies: CARE, NanoJ, NanoJ-Fluidics, NanoJ-SRRF, Nuclear-Pores as references and ZeroCostDL4Mic Funded by: BBSRC and Wellcome Trust News: ScienMag and EurekAlert! DOI: 10.1242/jcs.240713

Between life and death - strategies to reduce phototoxicity in super-resolution microscopy Kalina L Tosheva, Yue Yuan, Pedro Matos Pereira, Siân Culley, Ricardo Henriques Review published in Journal of Physics D - Applied Physics, January 2020 Technologies: CARE, NanoJ, NanoJ-Fluidics and NanoJ-SRRF Funded by: BBSRC and Wellcome Trust DOI: 10.1088/1361-6463/ab6b95

An Introduction to Live-Cell Super-Resolution Imaging Siân Culley, Pedro Matos Pereira, Romain F Laine, Ricardo Henriques Book chapter published in Imaging from Cells to Animals In Vivo, January 2020 Technologies: CARE, NanoJ, NanoJ-Fluidics, NanoJ-SQUIRREL, NanoJ-SRRF and QuickPALM DOI: 10.1201/9781315174662-4

Fluctuation-based super-resolution traction force microscopy Aki Stubb, Romain F Laine, Mitro Miihkinen, Hellyeh Hamidi, Camilo Guzmán, Ricardo Henriques, Guillaume Jacquemet, Johanna Ivaska Paper published in Nano letters, January 2020 Technologies: CARE, FBSR-TFM, NanoJ, NanoJ-SQUIRREL and NanoJ-SRRF Funded by: BBSRC DOI: 10.1021/acs.nanolett.9b04083

Nuclear pores as versatile reference standards for quantitative superresolution microscopy Jervis Vermal Thevathasan, Maurice Kahnwald, Konstanty Cieśliński, Philipp Hoess, Sudheer Kumar Peneti, Manuel Reitberger, Daniel Heid, Krishna Chaitanya Kasuba, Sarah Janice Hoerner, Yiming Li, Yu-Le Wu, Markus Mund, Ulf Matti, Pedro Matos Pereira, Ricardo Henriques, Bianca Nijmeijer, Moritz Kueblbeck, Vilma Jimenez Sabinina, Jan Ellenberg, Jonas Ries Paper published in Nature Methods, September 2019 Technologies: CARE, NanoJ, NanoJ-SQUIRREL, NanoJ-SRRF and Nuclear-Pores as references Funded by: BBSRC and Wellcome Trust News: Mirage News DOI: 10.1038/s41592-019-0574-9

Artificial intelligence for microscopy - what you should know Lucas von Chamier, Romain F. Laine, Ricardo Henriques Review published in Biochemical Society Transactions, July 2019 Technologies: CARE, NanoJ, NanoJ-Fluidics, NanoJ-SQUIRREL and NanoJ-SRRF Funded by: BBSRC and Wellcome Trust DOI: 10.1042/bst20180391

A hidden Markov model approach to characterizing the photo-switching behavior of fluorophores Lekha Patel, Nils Gustafsson, Yu Lin, Raimund Ober, Ricardo Henriques, Edward Cohen Published in The annals of applied statistics, January 2019 Technologies: CARE Funded by: BBSRC DOI: 10.1214/19-AOAS1240

Fix your membrane receptor imaging - actin cytoskeleton and CD4 membrane organization disruption by chemical fixation Pedro M Pereira, David Albrecht, Siân Culley, Caron Jacobs, Mark Marsh, Jason Mercer, Ricardo Henriques Published in Frontiers in immunology, January 2019 Technologies: CARE, NanoJ, NanoJ-Fluidics and NanoJ-SQUIRREL Funded by: BBSRC and Wellcome Trust DOI: 10.3389/fimmu.2019.00675

Content-aware image restoration - pushing the limits of fluorescence microscopy Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, Mauricio Rocha-Martins, Fabián Segovia-Miranda, Caren Norden, Ricardo Henriques, Marino Zerial, Michele Solimena, Jochen Rink, Pavel Tomancak, Loic Royer, Florian Jug, Eugene W. Myers Paper published in Nature Methods, November 2018 Technologies: CARE, NanoJ-SQUIRREL and NanoJ-SRRF Funded by: BBSRC and Wellcome Trust News: Technology Networks, VBIO, Innovations Report and Informationsdienst Wissenschaft DOI: 10.1038/s41592-018-0216-7

---

Funding contributing to CARE

SMALS - Smart Microscopy for Adaptative Live Super-resolution imaging to elucidate the initial steps of the HIV viral transmission Estibaliz Gómez-de-Mariscal Alias: SMALS Funded by: FCT - CEEC Individual Duration: April 2024 - March 2027

3D Nanoscope - a highly accessible, high-performance device for live cell nanoscopy Arturo G. Vesga Alias: 3DNanoScope4All Funded by: Marie Curie - HORIZON TMA MSCA Postdoctoral Fellowships Duration: March 2024 - February 2026

Sub-cellular Metabolic Compartmentalization During Oocyte Development Zita Carvalho dos Santos, Ricardo Henriques, Jorge Carvalho Funded by: CZI - Measuring Metabolism Across Scales Duration: January 2024 - December 2026

A transformative data-driven live-cell super-resolution microscopy development to elucidate the initial steps of effective viral transmission Estibaliz Gómez-de-Mariscal Funded by: EMBO - Postdoctoral Fellowships Duration: July 2022 - June 2024 Publications: 5

Real-Time high-content Super-Resolution Imaging of ES Cell States Eran Meshorer, Ricardo Henriques, Anna Kreshuk, Sandrine Lévêque-Fort, Nicolas Bourg, Genevieve Almouzni Alias: RT-SuperES Funded by: H2022 - EIC Pathfinder Open Duration: April 2022 - March 2027 Publications: 6

VP-CLEM-KIT - a pipeline for democratising volumetric visual proteomics Lucy Collinson, Ricardo Henriques, Paul French Funded by: CZI - Visual Proteomics Imaging Duration: December 2021 - June 2024 Publications: 11

Enabling Live-Cell 4D Super-Resolution Microscopy Guided by Artificial Intelligence Ricardo Henriques Alias: SelfDriving4DSR Funded by: ERC - Consolidator Duration: September 2021 - September 2026 Publications: 23

Mapping the early stages of HIV-1 infection by live-cell 4D Super-Resolution Microscopy Hannah Heil Funded by: EMBO - Postdoctoral Fellowships Duration: September 2021 - August 2023 Publications: 5

Artificial Intelligence for Image Data Analysis in the Life Sciences Anna Kreshuk, Florian Jug, Ricardo Henriques, Wei Ouyang, Arrate Muñoz-Barrutia, Emma Lundberg, Matthew Hartley Alias: AI4Life Funded by: H2021 - INFRA Duration: September 2021 - August 2025 Publications: 10

Optial Biology PhD programme Michael Hausser, Ricardo Henriques, Antonella Riccio Funded by: Wellcome Trust - 4-year PhD Programme in Science Duration: August 2021 - August 2025

Mapping HIV-1 infection by 4D Super-Resolution Microscopy Hannah Heil Funded by: FCT - CEEC Individual Duration: July 2021 - June 2027 Publications: 2

Unveiling live-cell viral replication at the nanoscale Ricardo Henriques Funded by: EMBO - Installation Grant Duration: January 2021 - January 2026 Publications: 16

Developing AI for Microscopy Ricardo Henriques Funded by: NVIDIA - NVIDIA Academic Hardware Grant Program Duration: December 2018 - December 2018