DeepTrack2 - A comprehensive deep learning framework for digital microscopy.

Installation • Getting Started • Examples • Advanced Tutorials • Developer Tutorials • Cite us • License

DeepTrack2 is a modular Python library for generating, manipulating, and analyzing image data pipelines for machine learning and digital microscopy.

TensorFlow Compatibility Notice: DeepTrack2 version 2.0 and subsequent do not support TensorFlow. If you need TensorFlow support, please install the legacy version 1.7.

The following quick-start guide is intended for complete beginners to understand how to use DeepTrack2, from installation to training your first model. Let's get started!

Installation

DeepTrack2 requires at least python 3.10.

To install DeepTrack2, open a terminal or command prompt and run:

pip install deeptrack

or

python -m pip install deeptrack

This will automatically install the required dependencies.

Getting Started

Here you find a series of notebooks providing an overview of the core features of DeepTrack2 and how to use them:

• DTGS101 Introduction to DeepTrack2

  Overview of how to use DeepTrack2. Creating images combining DeepTrack2 features, extracting properties, and using them to train a neural network.

• DTGS106 Simulating Different Image Modalities

  Simulating a spherical particle with different image modalities and generating a movie where this particle diffuses with passive Brownian motion.

• DTGS111 Loading Image Files Using Sources

  Using sources to load image files and to train a neural network.

• DTGS121 Tracking a Point Particle with a CNN

  Tracking a point particle with a convolutional neural network (CNN) using simulated particles resolved through a microscope with aberrations.

• DTGS126 Characterizing Aberrations with a CNN

  Characterizing spherical aberrations of an optical device with a convolutional neural network (CNN) using simulated images in the training process.

• DTGS127 Characterizing Aberrations with Optuna

  Characterizing aberrations of an optical device with the optimization framework Optuna.

• DTGS131 Tracking Multiple Particles with a U-Net

  Tracking multiple particles using a U-net trained on simulated images.

• DTGS141 Distinguishing Particles with a U-Net

  Tracking and distinguishing particles of different sizes in brightfield microscopy using a U-net trained on simulated images.

• DTGS151 Unsupervised Object Detection

  Single-shot unsupervised object detection using LodeSTAR.

• DTGS161 Fitting Using PyTorch Gradients

  Using PyTorch gradients to fit a Gaussian generated by a DeepTrack2 pipeline.

• DTGS171 Creating Custom Scatterers

  Creating custom scatterers of arbitrary shapes.

• DTGS172 Simulating Bacteria

  Creating custom scatterers in the shape of bacteria.

Examples

These are examples of how DeepTrack2 can be used on real datasets:

• DTEx211 MNIST

  Training a fully connected neural network to identify handwritten digits using MNIST dataset.

• DTEx212 Single Particle Tracking

  Tracks experimental videos of a single particle.

  
  

• DTEx213 Multi-Particle Tracking

  Detecting quantum dots in a low SNR image.

  
  

• DTEx214 Particle Feature Extraction

  Extracting the radius and refractive index of particles.

• DTEx215 Cell Counting

  Counting the number of cells in fluorescence images.

• DTEx216 3D Multi-Particle tracking

  Tracking multiple particles in 3D for holography.

• DTEx217 GAN image generation

  Using a GAN to create cell image from masks.

Specific examples for label-free particle tracking using LodeSTAR:

• DTEx231A LodeSTAR to Detect Particles

  
  

• DTEx231B LodeSTAR to Detect Particles of Various Shapes

• DTEx231C LodeSTAR to Measure the Mass of Particles in Holography

• DTEx231D LodeSTAR to Detect the Cells in the BF-C2DT-HSC Dataset

• DTEx231E LodeSTAR to Detect the Cells in the Fluo-C2DT-Huh7 Dataset

• DTEx231F LodeSTAR to Detect the Cells in the PhC-C2DT-PSC Dataset

• DTEx231G LodeSTAR to Detect Plankton

• DTEx231H LodeSTAR to Detect Particles in 3D Holography

• DTEx231I LodeSTAR Measure the Mass of Simulated Particles

• DTEx231J LodeSTAR to Measure the Mass of Cells

Specific examples for graph-neural-network-based particle linking and trace characterization using MAGIK:

• DTEx241A MAGIK to Trace Migrating Cells

  
  

• DTEx241B MAGIK to Track HeLa Cells

Advanced Tutorials

This section provides a list of advanced tutorials. The primary focus of these tutorials is to demonstrate the functionalities of individual modules and how they work in relative isolation, helping to provide a better understanding of them and their roles in DeepTrack2.

• DTAT301 deeptrack.features

• DTAT311 deeptrack.properties

• DTAT321 deeptrack.properties

• DTAT331 deeptrack.sequences

• DTAT351 deeptrack.utils

• DTAT353 deeptrack.statistics

• DTAT355 deeptrack.types

• DTAT357 deeptrack.elementwise

• DTAT361 deeptrack.sources.base

• DTAT362 deeptrack.sources.folder

• DTAT366 deeptrack.pytorch.data

• DTAT367 deeptrack.pytorch.features

• DTAT369 deeptrack.extras.radialcenter

• DTAT391 deeptrack.backend.core

• DTAT392 deeptrack.backend.pint_definition

• DTAT393 deeptrack.backend.units

• DTAT394 deeptrack.backend.polynomials

• DTAT395 deeptrack.backend.mie

• DTAT396 deeptrack.backend._config

Advanced Tutorials for Optical Microscopy

• DTATo10 deeptrack.optical.scatterers

• DTATo20 deeptrack.optical.optics

• DTATo30 deeptrack.optical.holography

• DTATo40 deeptrack.optical.aberrations

• DTATo50 deeptrack.optical.noises

• DTATo60 deeptrack.optical.augmentations

• DTATo90 deeptrack.optical.math

Developer Tutorials

Here you will find a series of notebooks tailored for DeepTrack2's developers:

• DTDV401 Overview of Code Base

• DTDV411 Style Guide

• DTDV421 Using Different Computational Backends

Documentation

The detailed documentation of DeepTrack2 is available at the following link: https://deeptrackai.github.io/DeepTrack2

Cite us!

If you use DeepTrack2 in your project, please cite us:

https://pubs.aip.org/aip/apr/article/8/1/011310/238663

"Quantitative Digital Microscopy with Deep Learning."
Benjamin Midtvedt, Saga Helgadottir, Aykut Argun, Jesús Pineda, Daniel Midtvedt & Giovanni Volpe.
Applied Physics Reviews, volume 8, article number 011310 (2021).

See also:

https://nostarch.com/deep-learning-crash-course

Deep Learning Crash Course
Benjamin Midtvedt, Jesús Pineda, Henrik Klein Moberg, Harshith Bachimanchi, Joana B. Pereira, Carlo Manzo & Giovanni Volpe.
2025, No Starch Press (San Francisco, CA)
ISBN-13: 9781718503922

https://www.nature.com/articles/s41467-022-35004-y

"Single-shot self-supervised object detection in microscopy." 
Benjamin Midtvedt, Jesús Pineda, Fredrik Skärberg, Erik Olsén, Harshith Bachimanchi, Emelie Wesén, Elin K. Esbjörner, Erik Selander, Fredrik Höök, Daniel Midtvedt & Giovanni Volpe
Nature Communications, volume 13, article number 7492 (2022).

https://www.nature.com/articles/s42256-022-00595-0

"Geometric deep learning reveals the spatiotemporal fingerprint of microscopic motion."
Jesús Pineda, Benjamin Midtvedt, Harshith Bachimanchi, Sergio Noé, Daniel Midtvedt, Giovanni Volpe & Carlo Manzo
Nature Machine Intelligence volume 5, pages 71–82 (2023).

https://doi.org/10.1364/OPTICA.6.000506

"Digital video microscopy enhanced by deep learning."
Saga Helgadottir, Aykut Argun & Giovanni Volpe.
Optica, volume 6, pages 506-513 (2019).

Funding

This work was supported by the ERC Starting Grant ComplexSwimmers (Grant No. 677511), the ERC Starting Grant MAPEI (101001267), and the Knut and Alice Wallenberg Foundation.