Journal of Physics D: Applied Physics ACCEPTED MANUSCRIPT • OPEN ACCESS NanoJ: a high-performance open-source super-resolution microscopy toolbox To cite this article before publication: Romain F. Laine et al 2019 J. Phys. D: Appl. Phys. in press https://doi.org/10.1088/1361-6463/ab0261 Manuscript version: Accepted Manuscript Accepted Manuscript is “the version of the article accepted for publication including all changes made as a result of the peer review process, and which may also include the addition to the article by IOP Publishing of a header, an article ID, a cover sheet and/or an ‘Accepted Manuscript’ watermark, but excluding any other editing, typesetting or other changes made by IOP Publishing and/or its licensors” This Accepted Manuscript is © 2019 IOP Publishing Ltd. As the Version of Record of this article is going to be / has been published on a gold open access basis under a CC BY 3.0 licence, this Accepted Manuscript is available for reuse under a CC BY 3.0 licence immediately. Everyone is permitted to use all or part of the original content in this article, provided that they adhere to all the terms of the licence https://creativecommons.org/licences/by/3.0 Although reasonable endeavours have been taken to obtain all necessary permissions from third parties to include their copyrighted content within this article, their full citation and copyright line may not be present in this Accepted Manuscript version. Before using any content from this article, please refer to the Version of Record on IOPscience once published for full citation and copyright details, as permissions may be required. All third party content is fully copyright protected and is not published on a gold open access basis under a CC BY licence, unless that is specifically stated in the figure caption in the Version of Record. View the article online for updates and enhancements. This content was downloaded from IP address 95.85.69.70 on 31/01/2019 at 08:50 ----!@#$NewPage!@#$---- NanoJ: a high-performance open-source super-resolution microscopy toolbox Romain F. Laine1-3 *, Kalina L. Tosheva1,2 *, Nils Gustafsson1,2,4, Robert D. M. Gray1,2,4, Pedro Almada1,2, David Albrecht1, Gabriel T. Risa1,6, Fredrik Hurtig7, Ann-Christin Lindås7, Buzz Baum1,6, Jason Mercer1, Christophe Leterrier5, Pedro M. Pereira1-3, Siân Culley1-3, and Ricardo Henriques1-3,6 1MRC-Laboratory for Molecular Cell Biology. University College London, London, UK 2Department of Cell and Developmental Biology, University College London, London, UK 3The Francis Crick Institute, London, UK 4Centre for Mathematics and Physics in Life Sciences and Experimental Biology (CoMPLEX), University College London, London, UK 5Aix-Marseille University, CNRS, INP, Institute of Neurophysiopathology, NeuroCyto, Marseille, France 6Institute for the Physics of Living Systems, University College London, London, UK 7Department of Molecular Biosciences, The Wenner-Gren Institute, Stockholm University, Stockholm, Sweden *These authors contributed equally. Abstract. Super-resolution microscopy has become essential for the study of nanoscale biological processes. This type of imaging often requires the use of specialised image analysis tools to process a large volume of recorded data and extract quantitative information. In recent years, our team has built an open-source image analysis framework for super-resolution microscopy designed to combine high performance and ease of use. We named it NanoJ - a reference to the popular ImageJ software it was developed for. In this paper, we highlight the current capabilities of NanoJ for several essential processing steps: spatio-temporal alignment of raw data (NanoJ-Core), super-resolution image reconstruction (NanoJ-SRRF), image quality assessment (NanoJSQUIRREL), structural modelling (NanoJ-VirusMapper) and control of the sample environment (NanoJ- Fluidics). We expect to expand NanoJ in the future through the development of new tools designed to improve quantitative data analysis and measure the reliability of fluorescent microscopy studies. ImageJ | Fiji | Super-resolution microscopy | Image analysis | Image quality assessment | Fluidics | Resolution | Quantitative imaging | Modelling | Virus | Vaccinia | Pox | Archaea | Sulfolobus acidocaldarius Correspondence: p.pereira@ucl.ac.uk,s.culley@ucl.ac.uk, r.henriques@ucl.ac.uk Page 1 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- Introduction. Fluorescence microscopy has been ubiquitously used in biological studies since its invention in the 20th century. It can reveal subcellular structures and interactions between specifically labelled molecules and allows the quantification of their dynamic behaviour in living cells1. Extraction of this biologically relevant quantitative information from fluorescence microscopy data typically requires digital image processing and analysis2. In recent years, Super-Resolution Microscopy (SRM) techniques have extended the spatial resolving power of fluorescence microscopy beyond the diffraction limit3–5. Most SRM techniques use large quantities of raw data, often reaching several gigabytes to generate a single super-resolution image, thus requiring specialised high- performance image analysis tools. Several SRM image processing packages are available, such as ThunderSTORM6, LAMA7 and SIMcheck8 but each of these is focused on a specific type of SRM modality. Here, we present NanoJ, a highly versatile set of image acquisition and analysis methods developed to improve the reliability and quantifiability of microscopy experiments, with a particular focus on the demands of live-cell SRM. NanoJ is available as a series of ImageJ-based plugins which can be used independently or concomitantly. NanoJ (Fig. 1) is comprised of the following modules: NanoJ- Core - general image correction tools including drift correction and channel registration, both based on cross- correlation analysis; NanoJ-SRRF - an analytical approach capable of extracting super-resolution data from a short sequence of diffraction-limited images, which can be acquired using most microscopes9,10; NanoJ- SQUIRREL - an algorithm to evaluate resolution and the presence of artefacts in super-resolution images11; NanoJ-VirusMapper - a single particle analysis method to generate nanoscale models of biological structures such as viruses12–14; NanoJ-Fluidics - a hardware and software framework to control fluidics devices, enabling automation of multiplexed experiments15. Thus, the NanoJ framework is capable of solving common imaging problems with broad biological applications and is compatible with a multitude of fluorescence microscope setups and experimental protocols. Fig. 1. NanoJ framework. Currently NanoJ consists of 5 modules dedicated to super-resolution imaging and analysis. The NanoJ framework. NanoJ has been designed to integrate with the popular ImageJ or Fiji image analysis software16,17 and is easily installed as a standard set of plugins. NanoJ is also fully open-source and user-friendly. The graphical user interfaces (GUIs) are straightforward to use and its routines can be easily integrated within larger image analysis pipelines through the ImageJ macro language. NanoJ is designed to be an accessible tool for both nonexpert users and developers. Each of its modules possess their own separate manual and documentation. NanoJ is implemented in both Java (https://www.java.com/) and OpenCL (https://www.khronos.org/opencl), the latter language being used for high-performance analysis of image data through the use of Graphical Processing Units (GPUs). To date, it encompasses four Java ARchive (JAR) packages (NanoJ-SRRF, NanoJ-SQUIRREL, NanoJ-VirusMapper, NanoJ-Fluidics) that all depend on a central package (NanoJ-Core). The core package hosts the libraries that enable high-performance GPU-based computing analysis and a set of basic image analysis helper methods. The modular nature of NanoJ means that its components can be updated independently, and the framework can be easily extended by appending new analytic packages. Page 2 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- NanoJ-Core: Drift Correction. Sample drift commonly occurs during the acquisition of SRM data, often as a result of gradual changes in the temperature of microscope components. Drift introduces motion blur artefacts and thereby a loss of resolution. While most modern microscopes have an active focus-lock device that stabilises the motion of the sample in the axial direction (minimising focal drift), the sample will still be prone to lateral movement (Fig. 2a). However, in the case where the raw data is made up of a sequence of consecutive frames acquired rapidly, as is common in SRM methods such as Single Molecule Localization Microscopy (SMLM)3,4 or fluctuation-based approaches9,18,19, this lateral drift can be estimated (Fig. 2b) and analytically corrected for each time frame via post-processing (Fig. 2c- d). NanoJ breaks the task of drift correction into two distinct parts: estimation, followed by translation. As a first step, NanoJ-Core estimates the linear drift between two images by calculating their cross-correlation matrix (CCM) (Fig. 2b). The location of the peak intensity in the CCM determines the linear shift between the two images, and precise sub-pixel accuracy is achieved by up-scaling the CCM using a bicubic spline interpolation. Depending on the type of acquisition, the reference frame can either be the first frame of the raw data or the immediately preceding frame. Fig. 2d shows the drift in a 100-frame dataset as measured with respect to the first frame. Once drift is estimated, the dataset can be directly corrected by analytically translating each individual frame using a bicubic spline interpolation (Fig. 2c). We estimated the performance of NanoJ-Drift correction (see Supp. Note 3 and Fig. S1a) at varying levels of signal-to-noise ratio (SNR) and established that the errors remain within 30 nm even at very low SNR and drop quickly below 10 nm for SNR > 6, appropriate for super-resolution microscopy. The interpolation process will, however, change the noise properties of the resulting dataset20, which can have an influence on further analysis requiring specific assumptions in noise properties such as Maximum Likelihood Estimation (MLE) single-molecule localization analyses. Fig. 2. Drift correction with NanoJ-Core. a) Composite image of two frames from a time-lapse dataset of the same field-of-view. An artificially large drift was applied computationally in order to make it visible for figure rendering. b) Cross-correlation map (CCM) between the two frames shown in a). The vector position of the maximum indicates the linear shift between the two frames. c) Overlay of the two frames after drift correction using NanoJ-Core. d) Vertical and horizontal drift curves obtained using NanoJ-Core from the 100-frame raw data. The two images shown in a) correspond to the frames 0 (green) and 97 (magenta) of the raw data. For the specific case of SMLM datasets with sparse blinking, there will only be a weak correlation across frames, as there is little observable structure conserved between consecutive time points. One common strategy to alleviate this low correlation is to add fiduciary landmarks to the sample, such as static fluorescent beads. As an option NanoJ-Core can temporally bin images within the dataset, thus increasing the correlation between frames and allowing their shift to be more accurately estimated21. Drift estimation in NanoJ differs from strategies applied by other SRM algorithms, such as ThunderSTORM6, by analysing unprocessed raw data instead of post- processed super-resolution reconstructions. This allows the estimation to be decoupled from the super-resolution image reconstruction algorithm and hence drift-corrected raw data can be analysed using a wider range of methods including SRRF9 and SOFI18. Furthermore, NanoJ-SRRF (as described in a following section) can import the drift curve (Fig. 2d) created by NanoJ-Core Drift Correction Page 3 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- and use this information directly during analysis without the need to pre-translate each frame in the raw dataset. NanoJ-Core: Channel registration. In multicolour fluorescence microscopy, images acquired in different spectral channels are often misaligned as a result of chromatic aberrations in the optical path and the use of different filter sets for each colour. This misalignment is frequently ignored in conventional microscopy as it typically occurs on a scale smaller than the diffraction limit. However, this effect becomes nonnegligible in the context of SRM22. Channel registration is therefore essential for multicolour SRM studies quantifying colocalization or interactions between different structures23–25. Advanced imaging research groups tend to develop their own analysis scripts for their particular applications, and thus no user-friendly tools are readily available. Therefore, NanoJ-Core Channel registration offers a unique tool to perform this registration easily and robustly on a wide variety of datasets. The shift between different spectral channels is usually inhomogeneous across a field of view, which prevents the use of typical CCM-based approaches such as the one described for NanoJ-Core Drift Correction. In order to characterise the spectral misalignment across a field of view, it is necessary to image a sample where the same structure can be observed across all spectral channels of interest. Furthermore, the sample should contain structures occupying the whole field of view. A typical sample for this characterisation is a coverslip coated with a large number of beads labelled with multiple fluorescent dyes26. Following multicolour imaging of this sample, NanoJ-Core can be used to calculate a nonlinear two- dimensional spatial transform describing the misalignment for each channel relative to a reference channel. Given that chromatic misalignment is a fixed property of an optical system, NanoJ-Core can then apply these transforms to realign other multicolour datasets acquired using the same optical path (Fig. 3)27,28. To generate these non-linear misalignment fields NanoJ- Core first calculates local, linear misalignments between channels (Fig. 3a). This is achieved by dividing the image into small areas ("blocks"). For each block, the shift is calculated by finding the cross-correlation peak position as shown in Fig. 2a, b. These local shift values are then interpolated across the whole field-of-view using an inverse distance weighting interpolation29. This generates two smooth shift maps describing the misalignments in the horizontal and vertical directions (Fig. 3b). For a given channel, each pixel value within the horizontal/vertical shift map indicates the horizontal/vertical displacement that needs to be applied to that pixel to align it with the reference channel. NanoJ-Core can then be used to apply these maps to align channels in the dataset of interest (Fig. 3), provided they have been acquired using the same optical configuration. Here, we measured the Target Registration Error28,30 (TRE, see Supp. Note 3 and Fig. S1b) in typical multi-colour configurations and showed that the TRE remains below 50 nm for all combinations of channels. NanoJ-Core performs the channel registration by creating a new image representing each channel, where the intensity value for each pixel coordinate corresponds to the intensity value from the original image at the equivalent coordinate corrected for local shift. For cases in which these coordinates are not discrete (sub-pixel shift), a bicubic spline interpolation is used to recover pixel values in continuous space. Because the shift map can be extrapolated to continuous space, the registration procedure obtained from diffraction-limited images can also easily be used to correct super-resolved images obtained using the same optical configuration. NanoJ-SRRF: Live-Cell Super-Resolution Imaging. As part of the NanoJ framework, we include our recently developed SRM reconstruction algorithm Super- Resolution Radial Fluctuations (SRRF), which is able to extract sub-diffraction information from a short burst of images acquired at highspeed with modern fluorescence microscopes9,10. SRRF is a purely analytical approach. It alleviates the need to use toxic photoswitching-inducing buffers31, specialised fluorophores32,33, damaging high- Page 4 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- intensity illumination34 or specialised equipment5,35 when compared to other SRM methods 3–5,35. Fig. 3. Multi-colour channel registration with NanoJ- Core. a) Composite image of multi-colour TetraSpeck™ beads imaged in two different channels ("GFP-channel" indicated in green and "mCherry-channel" in magenta), prior to (left) and after (right) channel registration using NanoJ-Core. Insets - individual beads from indicated locations. Scale bars: 25 µm, insets: 0.5 µm. b) Vectorial representation of the shift between the two channels (left, displacement vector length 50 times larger for representation purposes), horizontal (middle) and vertical (right) shift maps obtained and applied to the data shown in a). Scale bars: 25 µm. SRRF is based on similar principles to SMLM, with the key difference that it does not rely on the detection of spatiotemporally isolated fluorophores. Instead, SRRF generates a magnified pixel grid where each pixel value relates to the probability of fluorophores existing in that corresponding region of space. To do this, SRRF calculates the local radial symmetry (’radiality’) in each pixel of the magnified image using local intensity gradient information. The obtained radiality value will be high when a point-spread-function (PSF) profile transiently becomes dominant, highlighting the presence of a fluorescent molecule at that location. Furthermore, the fluctuations of radiality values follow the underlying natural intensity fluctuations of fluorophores, which have a distinct temporal signature to that of noise18. Therefore, a temporal correlation of the radiality at each pixel can be projected into a final image, where the structures of interest will be better resolved. Thanks to its low-illumination requirement, SRRF is highly suited to enable live-cell SRM and allows for long SRM timelapse acquisitions10. Fig. 4 shows a typical live-cell SRRF acquisition of a Cos7 cell expressing UtrCH-GFP (a probe for actin filaments) imaged at 33.3 Hz. The dataset shown is part of a longer > 30 min time-course dataset (Sup. Movie 1 will be added soon). SRRF allows the observation of protein dynamics (e.g. actin) during long periods with no perceivable phototoxicity. Fig. 4. Live-cell super-resolution microscopy with NanoJ-SRRF. a) Comparison of widefield (left) and SRRF reconstruction (right) obtained from a Cos7 cell expressing UtrCH-GFP to label actin filaments. Scale bar: 5 µm. b) Time-course of the inset shown in a), obtained from a continuous imaging at 30 ms exposure (33.3 Hz) and displayed every 30 s. Scale bar: 1 µm. c) Colour- coded time course dataset from b). Scale bar: 1 µm. NanoJ-SRRF, the software implementation of the SRRF algorithm, uses GPU computing whenever possible to accelerate radial symmetry calculations. This is achieved by implementing part of its analytic engine in OpenCL, with a fallback of execution to the CPU if a compatible graphics card is not detected. Page 5 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- NanoJ-SQUIRREL: Estimating Image Quality. SRM techniques are more complex than conventional diffraction-limited microscopy. This complexity arises from the sample preparation requirements (SMLM), the microscope hardware (stimulated emission depletion microscopy or STED, Structured Illumination Microscopy or SIM) and the post-acquisition image processing (SMLM, SIM). Unsuitable choices in any of these underlying variables can lead to artefacts within the final image and the potential for false conclusions to be drawn. NanoJ- SQUIRREL (Super-resolution Quantitative Image Rating and Reporting of Error Locations) is an algorithm that highlights the presence of artefacts in super-resolution images. It does so by calculating quantitative maps showing both local SRM image quality and resolution11 and can thus be used to optimise acquisition workflows10. The central concept of SQUIRREL is that a diffraction- limited image and the corresponding SRM image of the same region should contain the same underlying structure, just at different resolutions. Thus, the diffraction-limited image can be treated as a high-confidence benchmark against which the SRM image can be compared. The SQUIRREL image quality assessment algorithm formalises this comparison analytically. NanoJ-SQUIRREL quality assessment requires the user to provide an acquired diffraction-limited image and the SRM image of the same structure (Fig. 5a). As both images represent the underlying fluorophore distribution but at different resolutions, there exists a blurring function that can convert the SRM image into its diffraction-limited equivalent. SQUIRREL calculates this blurring function and applies it to the SRM image. The blurred image is then compared against the diffraction-limited reference. This generates three quality indicators. An error map is produced that maps the discrepancy between the blurred SRM image and the reference image at every pixel (Fig. 5b). This highlights regions where the super-resolution image is inconsistent with the reference image. For example, the three insets in Fig. 5b show typical SMLM artefacts including incomplete structures and interstructure mislocalization. Two global quality metrics are also generated for each image: the RSP (Resolution Scaled Pearson’s Correlation Coefficient) and the RSE (Resolution Scaled Error). The RSP can take a value in the interval [-1,1] and describes the structural agreement between the blurred super-resolution and reference images. Here, higher values indicate better agreement, with an RSP of 1 indicating a perfect structural match. The RSE describes the mean intensity mismatch between the blurred super-resolution and reference images; in this case lower values represent better agreement with a value of 0 indicating a perfect intensity match. NanoJ-SQUIRREL: Estimating Image Resolution. The purpose of SRM is to resolve finer structural detail than is achievable with conventional diffraction-limited microscopy. It is therefore useful to have an objective measurement of resolution within a super-resolution image, for example to enable meaningful analysis and structural hypotheses that stay in line with the actual precision of the data. The current standard for measuring image resolution in SMLM images is Fourier Ring Correlation (FRC)36. This method involves comparing two independently acquired super-resolution images of the same field-of-view so that they only differ by their noise components. For SMLM datasets the two SRM images are usually obtained by splitting localizations from odd and even frames. The correlation between these two SRM images is measured at different frequencies in Fourier space; the frequency at which this correlation drops below a set threshold indicates the resolution of the image. FRC has been previously implemented in ImageJ36, but only gives a single resolution measurement for the entire field of view. However, resolution is not necessarily homogeneous across the SRM image (Fig. 5c). This is particularly true for SMLM methods as localization accuracy depends strongly on labelling density and laser illumination intensity, which can both vary considerably within a single field of view. Page 6 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- Fig. 5. Quality assessment and resolution mapping with NanoJ-SQUIRREL. a) A super-resolution rendering (left) and acquired widefield image (right) of fixed microtubules labelled with Alexa Fluor-647. b) Left: SQUIRREL error map highlighting discrepancies between the super-resolution and diffraction-limited images in (a). Right: Magnified insets of super-resolution rendering at indicated positions on error map. c) Left: SQUIRREL resolution map of the super-resolution image in (a). Right: Magnified insets of super-resolution rendering for indicated resolution blocks. Whole image scale bars = 5µm, inset scale bars = 1µm. Furthermore, FRC can generate biased measurements for certain fluorophore distributions such as point-like patterns. Therefore, an additional feature of the NanoJ-SQUIRREL plugin is local mapping of FRC resolution across an image. To do this, the user provides an image stack comprising two independent renderings of the same dataset (e.g. through the odd/even frames splitting as described above). The images are then spatially split into equal-sized blocks and FRC analysis is run locally on each block. For blocks where there is insufficient correlation to generate an FRC resolution value, a resolution value is interpolated from neighbouring blocks. Fig. 5c shows the FRC map obtained from the SRM image shown in (a). This map highlights that the resolution in this image varies between 85 and 36 nm. It is important to note that high resolution (that is, a low FRC value) does not imply that the super-resolution image has depicted structures correctly; it only means that there is low variation in the locations of the fluorophores between the two rendered images. Therefore, the error mapping functionality within NanoJ-SQUIRREL can complement FRC-mapping in order to obtain a more complete perspective on super-resolution image quality. NanoJ-VirusMapper: Structural Mapping and Modelling. As part of the NanoJ framework, we include a unique single-particle analysis (SPA) tool called NanoJ- VirusMapper. It is the first open-source, freely available algorithm for unbiased, high-throughput SPA of fluorescence imaging and allows the structural modelling of viruses and other macromolecular complexes 12–14. The principle of SPA is to image many identical copies of a structure, independently of its orientation, and align and combine them to build an averaged structural map of the underlying structure with high signal-to-noise ratio37–41. The SPA implementation of VirusMapper facilitates automatic processing of multiple images to detect, segment, align, classify and average thousands of individual structures. It is entirely general, assuming no underlying symmetry or other properties of the imaged structure. Here, we illustrate this with models of three distinct vaccinia virus substructures42: core, lateral bodies (LBs) and membrane. All three substructures were labelled on viral particles and imaged with SIM (Fig. 6a, top). VirusMapper can explicitly incorporate information from multiple fluorescence channels to enable multi- component modelling. Here, the images of the LBs and the core were used to find the orientations of each particle in the dataset. This allowed us to create simultaneous models of the three components of the virus (Fig. 6a, Page 7 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- bottom). VirusMapper can also be used to separately model different 3D orientations of a structure. Fig. 6. Quantitative SPA-based modelling with NanoJ- VirusMapper. a) Top: Aligned SIM images of individual vaccinia particles labelled for L4 (core), F17 (lateral bodies, LBs) and A17 (membrane, mem.). Bottom: VirusMapper models of the three channels. Scale bars: 200nm. b) Top: Aligned SIM images of individual Sulfolobus acidocaldarius cells labelled for the S-layer (S) and the archaeal ESCRT-III homolog CdvB (CdvB). Bottom: VirusMapper models of three different orientations of the cells; magenta - S-layer, green - CdvB. Scale bars: 1µm. Here, we highlight this capability on a study of the division ring in the thermoacidophilic archaeon Sulfolobus acidocaldarius, strain DSM639 (Fig. 6b). Sulfolobus cells were labelled for both their outer S-layer and the division ring, as marked by the archaeal ESCRT-III homolog CdvB. The images of the division ring were used to identify different orientations of the cells (Fig. 6b) and separate templates were created using VirusMapper. Parallel models were then generated of the division ring and the Slayer using these templates for each orientation. The models obtained from the two channels can then be overlayed (Fig. 6b, bottom). VirusMapper is compatible with any fluorescence microscopy method and has been demonstrated with SIM, STED microscopy12 and SMLM (Fig. S1). The general approach of the method enables it to even be applied to correlative combinations of methods (Fig. S1). NanoJ-Fluidics: Sample Liquid Exchange. NanoJ- Fluidics is a hardware and software framework for precise and accurate automated liquids exchange15. It was developed to enable automation of sample treatment and labelling of live or fixed specimens directly on the microscope stage15,43. The NanoJ-Fluidics hardware component is composed of customisable, low-cost and robust LEGO® syringe pumps and a liquid removal peristaltic pump, all controlled by simple Arduino® electronics. It is compatible with off the-shelf imaging chambers, without the need for any microfabrication. Its control software (the NanoJ-Fluidics module) is ImageJ- based and can be fully integrated with microscopy acquisition software. We have demonstrated the applicability of NanoJ-Fluidics in multiple experimental contexts, including in-situ correlative live-to-fixed super- resolution imaging, multimodal super-resolution imaging and event-driven fixation15. The approach can also be easily extended to protocol optimisation (e.g. titrating antibody concentrations or adjusting imaging buffer composition) or liquid exchange protocols integrated with the imaging (e.g. drug delivery or automated event-driven fixation). Here, we demonstrate NanoJ-Fluidics by acquiring a high quality multicolour SMLM dataset, where STORM and DNA-PAINT44 imaging strategies are combined into a single workflow (Fig. 7a). This approach is particularly suited to multi-target imaging 32, which is difficult to achieve with standard sample preparation techniques due to the low number of suitable fluorophores available for SMLM microscopy. With NanoJ-Fluidics, we can seamlessly perform all labelling steps in an automated and reliable manner directly on the microscope stage. We Page 8 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- showcase this using a four-channel acquisition of actin with STORM, and mitochondria, vimentin and clathrin with DNA-PAINT (Fig. 7b). NanoJ-Fluidics’ highly customisable nature has already spawned several alternative designs from the community (https://github.com/HenriquesLab/NanoJ-Fluidics/wiki), which are in constant development. NanoJ-Fluidics makes imaging protocol automation readily available to researchers, hence improving not only the reliability and repeatability of the protocols, but also the range of protocols that are achievable. Fig. 7. Automated DNA-PAINT and STORM imaging. a) NanoJ-Fluidics workflow used for multi-color STORM and DNA-PAINT imaging. b) Left: 4-channel merge of STORM and DNA-PAINT with actin (red, STORM), mitochondria (green, DNAPAINT I1 strand), vimentin (yellow, DNA- PAINT I2 strand) and clathrin (cyan, DNAPAINT I3 strand). Right: Single-channel images from the inset. Scale bars: 2 µm. Discussion and Future Perspectives. The NanoJ framework provides a unique and comprehensive set of tools to support fluorescence imaging, from data acquisition and protocol optimisation to structural quantification. It offers powerful and user-friendly solutions to common pitfalls of image analysis such as drift correction and channel registration (NanoJ-Core). It extends the tools available for SRM to expert and novice users alike and aligns with the general aim of the community to extend SRM to live-cell applications, offering the first open-source analytical approach for live cell SRM imaging (NanoJ-SRRF). NanoJ offers a way to correlate dynamic live-cell data with high-resolution fixed cell data in an automated and highly reproducible manner (NanoJ- Fluidics). It incorporates an analytical method for the generation of accurate, high-content, molecular specific models (NanoJ-VirusMapper). Finally, the NanoJ framework provides tools to quantitatively assess SRM image quality by spatially mapping unbiased quality metrics, artefacts (errors) and FRC resolution (NanoJ- SQUIRREL), with the aim to improve standards in assessing and reporting microscopy data. The NanoJ framework gives access user-friendly yet robust analytical methods in tandem with new approaches to perform live-cell SRM experiments. A typical SRM live- cell imaging experiment using the NanoJ framework would entail acquiring a SRRF-compatible raw data set in any modern fluorescence microscope; performing drift correction and channel registration; followed by SRRF analysis for SRM image reconstruction; and finally, quality check using SQUIRREL. Used concomitantly in this manner, these tools can be used to address a large variety of biological questions and can be seamlessly repeated for imaging protocol optimisation and complemented using NanoJ-VirusMapper and/or NanoJ-Fluidics. Thus, the NanoJ-enabled scientific pipeline allows any user, regardless of the experience level, to seamlessly obtain SRM quantitative data of the highest scientific quality. The NanoJ framework was developed to facilitate access and expand the options of research groups using fluorescence microscopy. Hence, speed, reliability, performance and cross compatibility are of paramount importance. In this context, NanoJ has been designed for high-performance image analysis, using GPU computing, ensuring the quick processing of large data volumes45–49. Further, NanoJ’s modular and open-source nature and its integration within ImageJ allow it to be used in conjunction with other analysis software packages50–53 (49–52). We are continuously supporting, adapting and expanding the framework to include new approaches, such as 3D imaging. Page 9 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- We expect NanoJ to set the standard for useful, open- source, high performance methods for the whole microscopy community. Software and Hardware Availability. NanoJ follows opensource software and hardware standards. Each of its modules can be installed by enabling the corresponding code repository in Fiji or by following the instructions on the corresponding websites: • https://github.com/HenriquesLab/NanoJ-Core • https://github.com/HenriquesLab/NanoJ-SRRF • https://bitbucket.org/rhenriqueslab/NanoJ- SQUIRREL • https://bitbucket.org/rhenriqueslab/NanoJ- VirusMapper • https://github.com/HenriquesLab/NanoJ-Fluidics ACKNOWLEDGEMENTS First and foremost, we would like to thank the ImageJ and Fiji open-source development community. Their astonishing work has been the inspiration to the creation of NanoJ. There has been a large number of beta-testers and NanoJ users whose feedback helped us to continually update and improve its code, to the best of our capacity we keep an updated list of significant contributors in the website corresponding to each NanoJ module. We thank Prof. Ralf Jungmann at Max Planck Institute of Biochemistry Munich for reagents and advice. This work was funded by grants from the UK Biotechnology and Biological Sciences Research Council (BB/M022374/1; BB/P027431/1; BB/R000697/1; BB/S507532/1) (R.H., P.M.P. and R.F.L.), the UK Medical Research Council (MR/K015826/1) (R.H.), the Wellcome Trust (203276/Z/16/Z) (S.C., R.H and B.B.), Core funding to the MRC Laboratory for Molecular Cell Biology, University College London (MC_UU12018/7) (J.M.), the European Research Council (649101—UbiProPox) (J.M.) and the Centre National de la Recherche Scientifique (CNRS ATIP-AVENIR program AO2016) (C.L.). N.G. and R.D.M.G funded by the Engineering and Physical Sciences Research Council (EP/L504889/1). P.A. was supported by a PhD fellowship from the UK’s Biotechnology and Biological Sciences Research Council. Research by B.B. was supported by UCL, Cancer Research UK (C1529/A17343), BBSRC (BB/P001440/1) and MRC (MC_CF12266). K.L.T. and G.T.R. are supported by a 4-year MRC Research Studentship. D.A. is presently a Marie Curie fellow (Marie Sklodowska-Curie 750673). A.C.L. AND F.H. were supported by the Swedish Research Council, grant 621-20134685. AUTHOR CONTRIBUTIONS These contributions follow the Contributor Roles Taxonomy guidelines: https://casrai.org/credit/. Conceptualization: K.L.T, R.F.L., N.G., R.D.M.G, P.A., D.A., G.T.R., B.B., J.M., C.L., P.M.P., S.C., R.H.; Data curation: K.L.T, R.F.L., R.D.M.G, G.T.R., J.M., C.L., P.M.P., S.C., R.H.; Formal analysis: K.L.T, R.F.L., R.D.M.G, C.L., S.C.; Funding acquisition: A.C.L, B.B., C.L., R.H.; Investigation: K.L.T, R.F.L., R.D.M.G, G.T.R., C.L., P.M.P., S.C.; Methodology: R.F.L., N.G., R.D.M.G, P.A., J.M., C.L., P.M.P., S.C., R.H.; Project administration: R.F.L., B.B., J.M., C.L., P.M.P., S.C., R.H.; Resources: K.L.T, R.D.M.G, D.A., G.T.R., F.H., A.C.L., B.B., J.M., C.L., P.M.P., S.C.; Software: R.F.L., N.G., R.D.M.G, P.A., C.L., P.M.P., S.C., R.H.; Supervision: R.F.L., A.L., B.B., J.M., C.L., P.M.P., S.C., R.H.; Validation: K.L.T, R.F.L., N.G., R.D.M.G, P.A., D.A., C.L., P.M.P., S.C.; Visualization: K.L.T, R.F.L., R.D.M.G, D.A., C.L., P.M.P., S.C., R.H.; Writing – original draft: K.L.T, R.F.L., R.D.M.G, D.A., C.L., P.M.P., S.C., R.H.; Writing – review & editing: all authors. COMPETING FINANCIAL INTERESTS The authors declare no competing financial interests. Page 10 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- Bibliography 1. Rino, J., Braga, J., Henriques, R. & Carmo-Fonseca, M. Frontiers in fluorescence microscopy. Int. J. Dev. Biol. 53, 1569–1579 (2009). 2. Wheeler, A. & Henriques, R. Standard and Super-resolution Bioimaging Data Analysis: A Primer. (John Wiley & Sons., 2017). 3. Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution (SOM). Science (80-. ). x, 1642–1646 (2006). 4. Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–796 (2006). 5. Hell, S. W. & Wichmann, J. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy. Opt. Lett. 19, 780 (1994). 6. Ovesný, M. et al. ThunderSTORM: A comprehensive ImageJ plug-in for PALM and STORM data analysis and super- resolution imaging. Bioinformatics 30, 2389–2390 (2014). 7. Malkusch, S. & Heilemann, M. Extracting quantitative information from single-molecule super-resolution imaging data with LAMA – LocAlization Microscopy Analyzer. Sci. Rep. 6, (2016). 8. Schermelleh, L. et al. SIMcheck: a toolbox for successful super-resolution SIM imaging. Sci. Rep. 1–11 (2015). 9. Gustafsson, N. et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat. Commun. 7, 12471 (2016). 10. Culley, S., Tosheva, K. L., Matos Pereira, P. & Henriques, R. SRRF: Universal live-cell super-resolution microscopy. Int. J. Biochem. Cell Biol. 101, 74–79 (2018). 11. Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods (2018). doi:10.1038/nmeth.4605 12. Gray, R. D. M. et al. VirusMapper: open-source nanoscale mapping of viral architecture through super-resolution microscopy. Sci. Rep. 6, 29132 (2016). 13. Gray, R. D. M., Mercer, J. & Henriques, R. Open-source Single-particle Analysis for Super-resolution Microscopy with VirusMapper. J. Vis. Exp. 1–10 (2017). doi:10.3791/55471 14. Gray, R. et al. Nanoscale Polarization of the Vaccinia Virus Entry Fusion Complex Drives Efficient Fusion. bioRxiv 360073 (2018). doi:10.1101/360073 15. Almada, P. et al. Automating multimodal microscopy with NanoJ-Fluidics. bioRxiv 320416 (2018). doi:10.1101/320416 16. Abramoff, M. D., Magalhães, P. J. & Ram, S. J. Image Processing with ImageJ. Biophotonics Int. 11, 36–42 (2004). 17. Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676–682 (2012). 18. Dertinger, T., Colyer, R., Iyer, G., Weiss, S. & Enderlein, J. Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI). Proc. Natl. Acad. Sci. 106, 22287–22292 (2009). 19. Cox, S. et al. Bayesian localization microscopy reveals nanoscale podosome dynamics. Nat. Methods 9, 195–200 (2011). 20. Blaysat, B., Grédiac, M. & Sur, F. Effect of interpolation on noise propagation from images to DIC displacement maps. Int. J. Numer. Methods Eng. 108, 213–232 (2016). 21. Mlodzianoski, M. J. et al. Sample drift correction in 3D fluorescence photoactivation localization microscopy. Opt. Express 19, 15009 (2011). 22. Erdelyi, M. et al. Correcting chromatic offset in multicolor super-resolution localization microscopy. Opt. Express 21, 10978 (2013). 23. Bock, H. et al. Two-color far-field fluorescence nanoscopy based on photoswitchable emitters. Appl. Phys. B 88, 161– 165 (2007). 24. Van De Linde, S. et al. Multicolor photoswitching microscopy for subdiffraction-resolution fluorescence imaging. Photochem. Photobiol. Sci. 8, 465–469 (2009). 25. Niekamp, S., Sung, J., Huynh, W., Vale, R. D. & Stuurman, N. High accuracy measurements of nanometer-scale distances between fluorophores at the single-molecule level. bioRxiv 234740 (2017). doi:10.1101/234740 26. Demmerle, J. et al. Strategic and practical guidelines for successful structured illumination microscopy. Nat. Protoc. 12, Page 11 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- 988–1010 (2017). 27. Arganda-Carreras, I. et al. Consistent and Elastic Registration of Histological Sections Using Vector-Spline Regularization BT - Computer Vision Approaches to Medical Image Analysis. in (eds. Beichel, R. R. & Sonka, M.) 85– 95 (Springer Berlin Heidelberg, 2006). 28. Annibale, P., Scarselli, M., Greco, M. & Radenovic, A. Identification of the factors affecting co-localization precision for quantitative multicolor localization microscopy. Opt. Nanoscopy 1, 9 (2012). 29. Shepard, D. A two-dimensional interpolation for irregularly-spaced data function. Proc. Ass. Comput. Mach. 517–524 (1968). doi:10.1145/800186.810616 30. Churchman, L. S., Okten, Z., Rock, R. S., Dawson, J. F. & Spudich, J. A. Single molecule high-resolution colocalization of Cy3 and Cy5 attached to macromolecules measures intramolecular distances through time. Proc. Natl. Acad. Sci. 102, 1419–1423 (2005). 31. Henriques, R., Griffiths, C., Rego, E. H. & Mhlanga, M. M. PALM and STORM: Unlocking live-cell super-resolution. Biopolymers 95, 322–331 (2011). 32. Dempsey, G. T., Vaughan, J. C., Chen, K. H., Bates, M. & Zhuang, X. Evaluation of fluorophores for optimal performance in localization-based super-resolution imaging. Nat. Methods 8, 1027–1036 (2011). 33. Henriques, R. & Mhlanga, M. M. PALM and STORM: What hides beyond the Rayleigh limit? Biotechnol. J. 4, 846–857 (2009). 34. Wäldchen, S., Lehmann, J., Klein, T., van de Linde, S. & Sauer, M. Light-induced cell damage in live-cell super-resolution microscopy. Sci. Rep. 5, 15348 (2015). 35. Gustafsson, M. G. L. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. SHORT COMMUNICATION. J. Microsc. 198, 82–87 (2000). 36. Nieuwenhuizen, R. P. J. et al. Measuring image resolution in optical nanoscopy. Nat. Methods 10, 557–62 (2013). 37. Szymborska, A. et al. Nuclear pore scaffold structure analyzed by super-resolution microscopy and particle averaging. Science 341, 655–8 (2013). 38. Laine, R. F. et al. Structural analysis of herpes simplex virus by optical super-resolution imaging. Nat. Commun. 6, 5980 (2015). 39. Lelek, M. et al. Superresolution imaging of HIV in infected cells with FlAsH-PALM. Proc. Natl. Acad. Sci. 109, 8564– 8569 (2012). 40. Verdier, T., Gunzenhauser, J., Manley, S. & Castelnovo, M. Single particle maximum likelihood reconstruction from superresolution microscopy images. PLoS One 12, e0172943 (2017). 41. Sieben, C., Banterle, N., Douglass, K. M., Gönczy, P. & Manley, S. Multicolor single-particle reconstruction of protein complexes. Nat. Methods 15, 777–780 (2018). 42. Moss, B. in Fields Virology 6 ed. (eds. Knipe, D. M. & Howley, P.) 2664 (Philadelphia, USA: Lippincott Williams & Wilkins, a Wolters Kluwer business, 2013). 43. Dix, C. L. et al. The Role of Mitotic Cell-Substrate Adhesion Re-modeling in Animal Cell Division. Dev. Cell 45, 132– 145.e3 (2018). 44. Jungmann, R. et al. Multiplexed 3D cellular super-resolution imaging with DNA-PAINT and Exchange-PAINT. Nat. Methods 11, 313–8 (2014). 45. Herbert, S., Soares, H., Zimmer, C. & Henriques, R. Single-Molecule Localization Super-Resolution Microscopy: Deeper and Faster. Microsc. Microanal. 18, 1419–1429 (2012). 46. Pereira, P. M., Almada, P. & Henriques, R. High-content 3D multicolor super-resolution localization microscopy. Methods Cell Biol. 125, 95–117 (2015). 47. Almada, P., Culley, S. & Henriques, R. PALM and STORM: Into large fields and high-throughput microscopy with sCMOS detectors. Methods 88, 109–121 (2015). 48. Beghin, A. et al. Localization-based super-resolution imaging meets high-content screening. Nat. Methods 14, 1184– 1190 (2017). 49. Douglass, K. M., Sieben, C., Archetti, A., Lambert, A. & Manley, S. Super-resolution imaging of multiple cells by optimized flat-field epi-illumination. Nat. Photonics 10, 705–708 (2016). Page 12 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript ----!@#$NewPage!@#$---- 50. Sage, D. et al. Super-resolution fight club: A broad assessment of 2D & 3D single-molecule localization microscopy software 2. (2018). doi:10.1101/362517 51. Weigert, M. et al. Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy. 1–16 (2017). doi:10.1101/236463 52. Henriques, R. et al. QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ. Nat. Methods 7, 339–340 (2010). 53. Laine, R. F. et al. MiLeSIM: combining super-resolution and machine learning permits high-throughput virus structure analysis. bioRxiv (2018). Page 13 of 13 AUTHOR SUBMITTED MANUSCRIPT - JPhysD-118607.R1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Accepted Manuscript