RandomRotate/RandomFlip break image_coord alignment during training

[egoal]

Thanks for the great work! I've been using pt-v3m1-0-image and noticed a potential training-time alignment issue:

image configs (e.g. configs/nuscenes/semseg-pt-v3m1-0-image.py) apply RandomRotate and RandomFlip on point cloud, which only modify coord without updating image_coord.
image_coord is pre-computed in Dataset.get_data() before the transform pipeline.
this should silently break the alignment.

If this is true, a large fraction of training samples carry misaligned image features, which hurts classes relying on precise spatial correspondence.

[YilmazKadir]

Thanks for raising this. This is the same concern discussed in #14.

The important point is that image_coord is used as a per-point lookup into the image feature map, rather than as the projection of the augmented coord.

The relevant code are:

1. NuScenesDataset.get_data() projects each original LiDAR point into the cameras and stores an image_coord and image_mask for every point.

2. RandomRotate, RandomScale, RandomFlip, and RandomJitter modify the 3D coordinates but do not reorder the points. Therefore, row i in image_coord still belongs to point i.

3. assign_image_feat() samples the DINOv2 feature at the stored pixel coordinate and assigns it to the corresponding point. The resulting image_feat is then carried as a per-point feature and fused into the 3D decoder in point_transformer_v3m1_image.py.

Thus, a rotation or flip does not cause point i to receive another point’s image feature. The augmented point retains its original point-to-image correspondence.

On the other hand, image-space transforms such as resizing or horizontal flipping must update image_coord because they change the pixel grid. In contrast, recomputing image_coord after a 3D-only augmentation would sample a different physical image location. Therefore, the current implementation preserves the intended point-image association.