3D segmentation lightning with 2 channel (ConcatItemsd) and using RandCropByPosNegLabeld

[sepidehamiri]

Hello, dear Monai team,
Thank you for the great package that you provided for us.
I'm using Spleen 3D segmentation with MONAI with my dataset. My dataset has two inputs (ct and pet), and I have one mask. I tried to concact ct and pet and use it as a two-channel input in U-net, but I got this error:
KeyError: 'image_petct'

Checking GPU availability
Available: True
Device count: 1
Current device: 0
Device name: NVIDIA GeForce RTX 3090
Device memory: 25438322688
Start training
/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/transforms/post/array.py:193: UserWarning: `to_onehot=True/False` is deprecated, please use `to_onehot=num_classes` instead.
  warnings.warn("`to_onehot=True/False` is deprecated, please use `to_onehot=num_classes` instead.")
/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:445: LightningDeprecationWarning: Setting `Trainer(gpus=[0])` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=[0])` instead.
  rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name          | Type     | Params
-------------------------------------------
0 | _model        | UNet     | 4.8 M 
1 | loss_function | DiceLoss | 0     
-------------------------------------------
4.8 M     Trainable params
0         Non-trainable params
4.8 M     Total params
19.239    Total estimated model params size (MB)
/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:219: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
Sanity Checking DataLoader 0: 100%|██████████| 1/1 [00:00<00:00,  1.57it/s]current epoch: 0 current mean dice: 0.0000
best mean dice: 0.0000 at epoch: 0
Epoch 0:   0%|          | 0/11 [00:00<?, ?it/s] /home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1891: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=16). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  rank_zero_warn(
Traceback (most recent call last):
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/process.py", line 95, in <module>
    Unet_baseline().process()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/process.py", line 85, in process
    monai_unet.train_PETCT(data_dir=self.nifti_path)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/monai_unet.py", line 245, in train_PETCT
    trainer.fit(net)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 700, in fit
    self._call_and_handle_interrupt(
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 654, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 741, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
    results = self._run_stage()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
    return self._run_train()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1282, in _run_train
    self.fit_loop.run()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 269, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 174, in advance
    batch = next(data_fetcher)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 185, in __next__
    return self.fetching_function()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 264, in fetching_function
    self._fetch_next_batch(self.dataloader_iter)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 278, in _fetch_next_batch
    batch = next(iterator)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 558, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 570, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 100, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/transforms/transform.py", line 90, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/transforms/transform.py", line 54, in _apply_transform
    return transform(parameters)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/transforms/croppad/dictionary.py", line 857, in __call__
    image = d[self.image_key] if self.image_key else None
KeyError: 'image_petct'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/data/dataset.py", line 97, in __getitem__
    return self._transform(index)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/data/dataset.py", line 394, in _transform
    return self._post_transform(pre_random_item)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/data/dataset.py", line 330, in _post_transform
    item_transformed = apply_transform(_transform, item_transformed)
  File "/home/sepideh/Desktop/Papers/MICCAI_challenge/monai_unet/venv/lib/python3.8/site-packages/monai/transforms/transform.py", line 114, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x7f47becd9400>

Epoch 0:   0%|          | 0/11 [00:01<?, ?it/s]

Process finished with exit code 1

and this is my code:

class Net(pytorch_lightning.LightningModule):
    def __init__(self):
        super().__init__()
        self._model = UNet(
            dimensions=3,
            in_channels=2,
            out_channels=2,
            channels=(16, 32, 64, 128, 256),
            strides=(2, 2, 2, 2),
            num_res_units=2,
            norm=Norm.BATCH,
        )
        self.loss_function = DiceLoss(to_onehot_y=True, softmax=True, include_background=False, batch=True)
        self.post_pred = AsDiscrete(argmax=True, to_onehot=True, n_classes=2)
        self.post_label = AsDiscrete(to_onehot=True, n_classes=2)
        self.dice_metric = DiceMetric(include_background=False, reduction="mean", get_not_nans=False)
        self.best_val_dice = 0
        self.best_val_epoch = 0

    def forward(self, x):
        return self._model(x)

    def prepare_data(self, data_dir='/home/sepideh/Desktop/Papers/MICCAI_challenge/registration -3D Unet/nifti/FDG-PET-CT-Lesions'):
        # set up the correct data path
        images_pt = sorted(glob.glob(os.path.join(data_dir, 'PETCT*', '*', "SUV*")))
        images_ct = sorted(glob.glob(os.path.join(data_dir, 'PETCT*', '*', "CTres*")))
        images_sg = sorted(glob.glob(os.path.join(data_dir, 'PETCT*', '*', "SEG*")))

        data_dicts = [
            {"image_pt": image_name_pt, "image_ct": image_name_ct, "image_sg": image_name_sg}
            for image_name_pt, image_name_ct, image_name_sg in zip(images_pt, images_ct, images_sg)
        ]
        # val_files = data_dicts
        train_files, val_files = data_dicts[:-100], data_dicts[-100:]
        # set deterministic training for reproducibility
        set_determinism(seed=0)

        # define the data transforms
        train_transforms = Compose(
            [
                LoadImaged(keys=["image_pt", "image_ct", "image_sg"]),
                ScaleIntensityRanged(
                    keys=["image_ct"], a_min=-100, a_max=250,
                    b_min=0.0, b_max=1.0, clip=False,
                ),
                ScaleIntensityRanged(
                    keys=["image_pt"], a_min=0, a_max=15,
                    b_min=0.0, b_max=1.0, clip=False,
                ),
                ConcatItemsd(keys=["image_pt", "image_ct"], name="image_petct", dim=0),

                # AddChanneld(keys=["image_pt", "image_ct", "image_sg"]),
                # concatenate pet and ct channels
                EnsureChannelFirstd(keys=["image_petct", "image_sg"]),
                # Orientationd(keys=["image_petct", "image_sg"], axcodes="LAS"),
                # Spacingd(
                #     keys=["image_pt", "image_ct", "image_sg"],
                #     pixdim=(2, 2, 3),
                #     mode=("bilinear", "bilinear", "nearest"),
                # ),

                # user can also add other random transforms
                ToTensord(keys=["image_petct"]),

                # RandAffined(
                #     keys=["image_petct", "image_sg"],
                #     mode=("bilinear", "nearest"),
                #     prob=1.0,
                #     spatial_size=(64, 64, 64),
                #     rotate_range=(0, 0, np.pi / 15),
                #     scale_range=(0.1, 0.1, 0.1)),
                CropForegroundd(keys=["image_petct", "image_sg"], source_key="image_petct"),
                # randomly crop out patch samples from
                # big image based on pos / neg ratio
                # the image centers of negative samples
                # must be in valid image area
                RandCropByPosNegLabeld(
                    keys=["image_petct", "image_sg"],
                    label_key="image_sg",
                    spatial_size=(5, 5, 5),
                    pos=1,
                    neg=1,
                    num_samples=4,
                    image_key="image_petct",
                    image_threshold=0,
                ),

            ]
        )

        val_transforms = Compose(
            [
                LoadImaged(keys=["image_pt", "image_ct", "image_sg"]),
                # AddChanneld(keys=["image_pt", "image_ct", "image_sg"]),
                ScaleIntensityRanged(
                    keys=["image_ct"], a_min=-100, a_max=250,
                    b_min=0.0, b_max=1.0, clip=False,
                ),
                ScaleIntensityRanged(
                    keys=["image_pt"], a_min=0, a_max=15,
                    b_min=0.0, b_max=1.0, clip=False,
                ),
                ConcatItemsd(keys=["image_pt", "image_ct"], name="image_petct", dim=0),
                # concatenate pet and ct channels
                EnsureChannelFirstd(keys=["image_petct", "image_sg"]),

                # Spacingd(
                #  keys=["image_pt", "image_ct", "image_sg"],
                # pixdim=(2, 2, 3),
                # mode=("bilinear", "bilinear", "nearest"),
                # ),
                # Orientationd(keys=["image_pt", "image_ct", "image_sg"], axcodes="LAS"),


                CropForegroundd(keys=["image_petct", "image_sg"], source_key="image_petct"),

                ToTensord(keys=["image_petct"]),
            ]
        )
        # we use cached datasets - these are 10x faster than regular datasets
        self.train_ds = PersistentDataset(
            data=train_files[:10], transform=train_transforms, cache_dir='cache_dir'
        )
        self.val_ds = PersistentDataset(data=val_files[:10], transform=val_transforms, cache_dir='cache_dir')

    def val_dataloader(self):
        val_loader = DataLoader(
            self.val_ds[:20], batch_size=1, num_workers=0, collate_fn=list_data_collate)
        return val_loader

    def train_dataloader(self):
        train_loader = DataLoader(
            self.train_ds, batch_size=12, shuffle=True,
            num_workers=16, collate_fn=list_data_collate,
        )
        return train_loader

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self._model.parameters(), 1e-4)
        return optimizer

    def training_step(self, batch, batch_idx):
        images, labels = batch["image_petct"], batch["image_sg"]
        output = self.forward(images)
        loss = self.loss_function(output, labels)
        tensorboard_logs = {"train_loss": loss.item()}
        return {"loss": loss, "log": tensorboard_logs}

    def validation_step(self, batch, batch_idx):
        images, labels = batch["image_petct"], batch["image_sg"]
        roi_size = (160, 160, 160)
        sw_batch_size = 4
        outputs = sliding_window_inference(
            images, roi_size, sw_batch_size, self.forward)
        loss = self.loss_function(outputs, labels)
        outputs = [self.post_pred(i) for i in decollate_batch(outputs)]
        labels = [self.post_label(i) for i in decollate_batch(labels)]
        self.dice_metric(y_pred=outputs, y=labels)
        return {"val_loss": loss, "val_number": len(outputs)}

    def validation_epoch_end(self, outputs):
        val_loss, num_items = 0, 0
        for output in outputs:
            val_loss += output["val_loss"].sum().item()
            num_items += output["val_number"]
        mean_val_dice = self.dice_metric.aggregate().item()
        self.dice_metric.reset()
        mean_val_loss = torch.tensor(val_loss / num_items)
        tensorboard_logs = {
            "val_dice": mean_val_dice,
            "val_loss": mean_val_loss,
        }
        if mean_val_dice > self.best_val_dice:
            self.best_val_dice = mean_val_dice
            self.best_val_epoch = self.current_epoch
        print(
            f"current epoch: {self.current_epoch} "
            f"current mean dice: {mean_val_dice:.4f}"
            f"\nbest mean dice: {self.best_val_dice:.4f} "
            f"at epoch: {self.best_val_epoch}"
        )
        return {"log": tensorboard_logs}


def train_PETCT(data_dir='/home/sepideh/Desktop/Papers/MICCAI_challenge/registration -3D Unet/nifti/FDG-PET-CT-Lesions', log_dir='model_log'):
    # initialise the LightningModule
    net = Net()
    # net.prepare_data(data_dir)
    # set up loggers and checkpoints
    log_dir = os.path.join(log_dir, "logs")
    tb_logger = pytorch_lightning.loggers.TensorBoardLogger(
        save_dir=log_dir
    )

    # initialise Lightning's trainer.
    trainer = pytorch_lightning.Trainer(
        gpus=[0],
        max_epochs=800,
        logger=tb_logger,
        enable_checkpointing=True,
        num_sanity_val_steps=1,
        log_every_n_steps=16,
    )

    # train
    trainer.fit(net)
    print(
        f"train completed, best_metric: {net.best_val_dice:.4f} "
        f"at epoch {net.best_val_epoch}")

[Nic-Ma]

Hi @sepidehamiri ,

I think the error is KeyError: 'image_petct' in the ToTensord(keys=["image_petct"]) transform?

Thanks.

[sepidehamiri]

Thank you for your help.
The problem was solved by changing ToTensord to EnsureTyped, but now I have another error:

    random_int = rand_state.randint(len(indices_to_use))
  File "mtrand.pyx", line 748, in numpy.random.mtrand.RandomState.randint
  File "_bounded_integers.pyx", line 1247, in numpy.random._bounded_integers._rand_int64
ValueError: high <= 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
.
.
.
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x7f97e46e2850>

[wxr521314]

hi, did you solve this problem? I had the same problem.