RFP/GFP/DIC Image Processing Pipeline @ EMBL

This document describes a Nextflow pipeline for registration, segmentation, and quantitative analysis of multi‑channel time‑lapse microscopy data (RFP, GFP, DIC) on a SLURM cluster with conda and optional Ilastik and YOLO‑based protist analysis.

---

Overview

The pipeline processes raw ND2/TIFF images into a structured project directory and performs:

• Grouping of raw images per experiment/chip/position.
• Conversion to OME‑TIFF stacks ((T, Z, C, Y, X)) with optional CLAHE and compression.
• Rigid registration (per chip) based on a reference position, and application to all channels.
• Per‑channel max‑projection and GFP spot detection using a normalized Difference‑of‑Gaussians (DoG) workflow.
• RFP segmentation via Ilastik, measurement of RFP objects, and merging of GFP/RFP measurements.
• DIC protist segmentation using YOLO, best z‑plane selection via tracking and focus metrics, and clustering/UMAP visualization.

---

Workflow visualization

Directory structure

By default, results are written to project/ inside the repository, with one subfolder per major step:

• 01_image_grouping/ – text files listing raw images grouped by experiment/chip/position.
• 02_process_groups/ – processed OME‑TIFF stacks ((T, Z, C, Y, X)).
• 03_registration_mats/ – .npy transformation matrices per chip for registration.
• 04_registered/ – registered OME‑TIFF stacks, including _zstack_registered.ome.tif used downstream.
• 05_max_projections/ – per‑channel max projections _channelX_maxproj.ome.tif.
• 06_gfp_segmentation/ – GFP masks and per‑timepoint measurements/summary CSVs.
• 07_rfp_segmentation/ – Ilastik segmentation maps for RFP.
• 08_rfp_measurements/ – RFP object measurement CSVs.
• 09_combined_results/ – merged GFP/RFP measurement tables combined_*.csv.
• 10_DIC_segmentation/ – YOLO predictions (predictions_raw/), cropped protist images (objects/), and per‑group metadata CSVs.
• 11_DIC_plane_selection/ – per‑stack best‑focus CSVs and visualizations + merged best_focus_objects.csv.
• 12_DIC_clustering/ – clustering results, CSVs, and visualizations (e.g. UMAP plots, HTML).

In addition, standard Nextflow reports are written to:

• project/timeline.html
• project/report.html
• project/trace.txt

---

Requirements

Software

• Nextflow (>= 22.04.0).
• SLURM cluster environment, with queues analogous to htc-el8 (CPU) and gpu-el8 (GPU) and suitable GPU options (e.g. --gpus 1 -C gpu=A100).
• Conda with at least two environments:
  • image_analysis – for registration, projections, GFP segmentation, RFP measurement, plane selection, clustering.
  • yolo – for YOLO‑based protist segmentation and some grouping/processing steps.
• Ilastik available as a module (e.g. module load ilastik) if RFP segmentation is used.

Python dependencies (indicative)

The Python scripts under bin/ require typical scientific Python stacks. Depending on your environment, install:

• Core: numpy, pandas, tifffile, tqdm, scikit-image, scipy, Pillow, opencv-python.
• GFP segmentation (segment_gfp_timeseries.py): numpy, pandas, tifffile, tqdm, Pillow, scikit-image, scipy.
• Protist segmentation (Protists_segmentation.py): numpy, opencv-python, pandas, ultralytics, scikit-image, tifffile, tqdm.

Data and models

• Raw ND2/TIFF data in a single input folder specified via --raw_dir.
• An Ilastik project .ilp if RFP segmentation is desired (--ilastik_project).
• A trained YOLO .pt model for protist segmentation (--yolo_model_path).

---

Installation

1. Obtain the pipeline

   Place main.nf, nextflow.config and the bin/ scripts in a project folder on the cluster (e.g. clone the repository or copy the files).

2. Set up conda environments

   Create two environments (adjust names/paths as you wish, but keep them in sync with nextflow.config):

   # Example: image_analysis env
   mamba create -n image_analysis python=3.10 \
       numpy pandas tifffile tqdm scikit-image scipy pillow opencv
   
   # Example: yolo env
   mamba create -n yolo python=3.10 \
       numpy pandas tifffile tqdm opencv ultralytics

3. Adjust paths in nextflow.config

   • Set workDir and process.scratch to your cluster’s scratch filesystem.
   • Update conda = '/home/USER/.conda/envs/...' entries to match your environment locations.
   • Adapt queue, clusterOptions and any GPU options to your SLURM configuration.

4. Ensure bin/ is referenced correctly

   params.scripts_dir defaults to "${baseDir}/bin", so the Python scripts must live in bin/ relative to main.nf.

---

Basic usage

Minimal full‑pipeline example:

nextflow run main.nf \
  --raw_dir /path/to/raw_nd2_or_tiffs \
  --project_dir /path/to/output_project \
  --ilastik_project /path/to/segmentation.ilp \
  --yolo_model_path /path/to/yolo_model.pt

Key points:

• --raw_dir is required and must point to the folder containing the raw ND2/TIFF files.
• --project_dir defines where the 01_… – 12_… subfolders and Nextflow reports will be created.
• --ilastik_project enables the RFP segmentation branch; if omitted, RFP segmentation/measurement is skipped with a warning.
• --yolo_model_path enables the protist DIC branch; if omitted, DIC segmentation, plane selection and clustering are skipped.

To print a concise built‑in help message:

nextflow run main.nf --help

---

Workflow summary

The Nextflow workflow block connects processes into two main branches: GFP/RFP and DIC protists, sharing the registration and max‑projection steps.

1. Input validation and logging

   • Fails fast if --raw_dir is not provided.
   • Logs all key parameters (project directory, Ilastik project path, register channels, GFP threshold, YOLO model, device and clustering settings) at startup.

2. 01 – GROUP_IMAGES

   • Runs group_raw_images.py on --raw_dir.
   • Produces:
     • *_files.txt – lists of raw files per logical group.
     • all_groups.txt – overview of all groups.

3. 02 – PROCESS_GROUPS

   • For each group list, runs process_groups_multipage.py.
   • Outputs OME‑TIFF stacks to 02_process_groups/ with shape roughly ((T, Z, C, Y, X)).
   • Controlled by:
     • params.multichannel_channels
     • params.multichannel_clahe
     • params.save_raw_with_clahe
     • params.proc_compression

4. 03 – CALC_REGISTRATION

   • Filters _zstack_processed.ome.tif files to only those with pos006 and runs calculate_registration_stackreg.py.
   • Writes chip‑specific transformation matrices as .npy into 03_registration_mats/.

5. 04 – APPLY_REGISTRATION

   • Applies the chip‑specific transformations to all z‑stack processed files using apply_registration_multicore.py.
   • If a chip has no transformation matrix (no suitable pos006), logs a warning and copies the input to a _registered file as a degenerate fallback.
   • Outputs _registered.ome.tif files in 04_registered/.

6. DIC branch (protists)

   • Filters registered stacks to those ending _zstack_registered.ome.tif.
   • 10 – PROTISTS_SEGMENT_DIC
     • Runs Protists_segmentation.py using YOLO on the DIC channel (channel index 2).
     • Produces:
       • 10_DIC_segmentation/predictions_raw/<group>/t###_z##.json with bounding boxes, classes, confidences and polygons.
       • 10_DIC_segmentation/objects/<group>/t###_z##/* with per‑object crops and masks.
       • 10_DIC_segmentation/metadata/<group>_objects.csv with geometric and intensity features.
   • 11 – PROTISTS_PLANE_SELECTION_STACK
     • For each registered stack, uses predictions + registered stacks to link detections across z and select best‑focus planes.
     • Outputs per‑stack <group>_best_focus_objects.csv, object crops and optional visualizations under 11_DIC_plane_selection/metadata and visualizations.
   • 11b – PROTISTS_PLANE_SELECTION_MERGE
     • Concatenates all per‑stack best‑focus CSVs into best_focus_objects.csv.
   • 12 – PROTISTS_CLUSTER_DIC
     • Runs protist_clustering.py with the merged best_focus_objects.csv, extracting features, applying PCA, KMeans clustering and generating UMAP/visualization outputs in 12_DIC_clustering/.

7. GFP/RFP branch

   • 05 – CALC_MAX_PROJECTIONS
     • Runs calculate_max_projections.py to produce _channelX_maxproj.ome.tif for each registered stack in 05_max_projections/.
   • 06 – SEGMENT_GFP
     • Uses segment_gfp_timeseries.py on _channel0_maxproj time series.
     • Workflow:
       • Normalizes each frame to ([0, 1]).
       • Applies Gaussian smoothing or DoG background subtraction.
       • Runs DoG blob detection with configurable min_sigma, max_sigma, sigma_ratio, threshold, overlap.
       • Filters small objects by min_object_size.
       • Saves label masks per timepoint and measurement/summary CSVs in 06_gfp_segmentation/.
   • 07 – SEGMENT_RFP_ILASTIK
     • For _channel1_maxproj images, runs segment_rfp_ilastik.py with the given Ilastik project and export source.
     • Generates _segmentation.tiff files in 07_rfp_segmentation/.
   • 08 – MEASURE_RFP_OBJECTS
     • Pairs each segmentation TIFF with the corresponding _maxproj.ome.tif intensity image.
     • Invokes measure_rfp_objects.py to compute shape and intensity features per RFP object in 08_rfp_measurements/.
   • 09 – COMBINE_MEASUREMENTS
     • Collects GFP CSVs and RFP CSVs, moves them into subfolders and runs combine_measurements.py.
     • Outputs combined_*.csv into 09_combined_results/.

---

Configuration (nextflow.config)

The main configuration file sets global defaults and per‑process overrides.

Global settings

• workDir – global Nextflow work directory on scratch.
• conda.enabled = true – enables automatic activation of conda envs defined per process.
• Global process defaults:
  • executor = 'slurm'
  • queue = 'htc-el8'
  • errorStrategy = 'finish'
  • maxRetries = 0
  • time = '1h'
  • cpus = 1
  • memory = '4.GB'
  • scratch = '/scratch/.../nextflow/'

Per‑process overrides (examples)

• GROUP_IMAGES
  • conda = '/path/to/envs/yolo', cpus = 2, memory = '2.GB', time = '10m'.
• PROCESS_GROUPS
  • Similar env, but memory = '16.GB', time = '20m', maxForks = 50.
• CALC_REGISTRATION, APPLY_REGISTRATION, CALC_MAX_PROJECTIONS, SEGMENT_GFP
  • Use image_analysis env with 1–2 CPUs, 8–16 GB RAM, and times from 30 min to a few hours.
• SEGMENT_RFP_ILASTIK
  • module = 'ilastik', conda = null, cpus = 8, memory = '32.GB', time = '4h', maxForks = 16.
• MEASURE_RFP_OBJECTS, COMBINE_MEASUREMENTS
  • 4 CPUs, 8–16 GB RAM, modest runtimes.
• PROTISTS_SEGMENT_DIC
  • conda = '/path/to/envs/yolo', cpus = 4, memory = '16.GB', time = '30m', queue = 'gpu-el8', clusterOptions = '--gpus 1 -C gpu=A100', maxForks = 4.
• PROTISTS_PLANE_SELECTION_STACK
  • cpus = 4, memory = '8.GB', time = '2h', maxForks = 50 for high parallelism across positions.
• PROTISTS_CLUSTER_DIC
  • cpus = 8, memory = '32.GB', time = '2h', maxForks = 1.

Reports and manifest

• timeline, report, trace blocks enable Nextflow HTML and text reports under ${params.project_dir}.
• manifest includes:
  • name = 'RFP/GFP/DIC Image Processing Pipeline @ EMBL'
  • author = 'Jonas Wagner'
  • Human‑readable description, version, and nextflowVersion = '>=22.04.0'.

---

Key parameters and how to tune them

All parameters below can be overridden on the command line as --param value.

Project and I/O

• --raw_dir (required): path to raw ND2/TIFF files; controls whether GROUP_IMAGES runs.
• --project_dir (default "${baseDir}/project"): output root for all step folders and reports.
• --scripts_dir (default "${baseDir}/bin"): folder containing the Python scripts.

Image grouping and processing

• --grouping_pattern (default "*.nd2"): pattern to match raw files.
• --multichannel_channels (default "1 2 3"): channels to extract into the multichannel stack.
• --multichannel_clahe (default "3"): channels to which CLAHE is applied.
• --save_raw_with_clahe (default "false"): whether to keep both raw and CLAHE versions.
• --proc_compression (default "none"): TIFF compression (none | deflate | lzw).

Registration

• --registration_channel (default 1): 0‑indexed channel used for registration (e.g. channel 2).
• --registration_reference (default "first"): "first" or "previous" frame as reference.
• --register_channels (default "0 1 2"): channels to which registration is applied.
• --apply_compression (default "none"): compression for registered OME‑TIFF output.

GFP segmentation (channel 0)

• --gfp_preprocess_method (default "dog_background"): "gaussian" or "dog_background" preprocessing.
• --gfp_sigma_smooth (default 1.0): Gaussian sigma when using gaussian.
• --gfp_min_sigma, --gfp_max_sigma (default 1.0, 2.0): DoG blob scale range.
• --gfp_sigma_ratio (default 1.2): spacing of scales in DoG.
• --gfp_threshold (default 0.02): detection threshold on normalized ([0, 1]) image; lower for more sensitive detection.
• --gfp_overlap (default 0.6): maximum allowed overlap between blobs.
• --gfp_min_object_size (default 4): minimum area (pixels) of kept objects.

RFP segmentation (channel 2)

• --ilastik_project (default null): path to Ilastik .ilp file; required for RFP segmentation.
• --ilastik_export_source (default "Simple Segmentation Stage 2"): Ilastik export source name.
• --ilastik_max_forks (default 8): maximum parallel Ilastik jobs.

Protist (DIC) segmentation and clustering

• --yolo_model_path (default set in code, but should be overridden): YOLO .pt model for DIC protist segmentation.
• --protist_device (default '0'): GPU index or 'cpu'.
• --protist_conf (default 0.2): YOLO confidence threshold.
• --protist_iou (default 0.1): YOLO IoU threshold for NMS.
• --protist_imgsz (default 1024): input size for YOLO.

Best‑plane selection parameters:

• --protist_iou_threshold (default 0.2): bbox IoU threshold for linking across z.
• --protist_centroid_dist (default 15): maximum centroid distance (pixels) for linking.
• --protist_max_z_gap (default 10): maximum z‑gap allowed in tracks.

Feature‑extraction flags:

• --protist_use_glcm (default "true"): toggle GLCM features.
• --protist_use_lbp (default "true"): toggle LBP features.
• --protist_use_hog (default "true"): toggle HOG features.

Clustering/embedding:

• --protist_n_clusters (default 3): KMeans cluster count.
• --protist_pca_components (default 5): PCA dimension.
• --protist_umap_neighbors (default 15): neighbors for UMAP.
• --protist_max_hover_images (default 20000): cap on number of images for interactive plots.

---

Cluster and performance tuning

• Adjust maxForks per process (e.g. PROCESS_GROUPS, SEGMENT_RFP_ILASTIK, MEASURE_RFP_OBJECTS, PROTISTS_SEGMENT_DIC, PROTISTS_PLANE_SELECTION_STACK) to match your cluster’s concurrency limits and memory.
• Increase time and memory in nextflow.config for large datasets or if jobs hit resource limits.
• On GPU nodes, adapt queue and clusterOptions to your scheduler (e.g. different GPU type, partition name, or number of GPUs).
• The trace.txt, timeline.html and report.html outputs are useful for profiling and further tuning.

---

Running only subsets of the pipeline

The current workflow always executes both branches when their prerequisites are present:

• To skip RFP: omit --ilastik_project. The pipeline logs a warning and skips Ilastik + RFP measurement + combination.
• To skip protists: omit --yolo_model_path. DIC segmentation, plane selection and clustering are skipped.

If you frequently need partial workflows (e.g. protists‑only), you can adapt workflow {} and when: clauses to guard certain branches behind additional flags.

---

Summary

This pipeline provides an end‑to‑end solution for RFP/GFP/DIC time‑lapse microscopy, including registration, spot detection, RFP quantification and GPU‑accelerated protist segmentation, best‑focus selection and clustering. With appropriate conda environments and SLURM configuration, it can be used as a general‑purpose image‑analysis backbone that you can further specialize by tweaking parameters in main.nf and nextflow.config.