Skip to content

Add hierarchical track classification - #1847

Open
PaulHax wants to merge 6 commits into
clone-metadata-copyfrom
hierarchical-classification-pr1
Open

Add hierarchical track classification#1847
PaulHax wants to merge 6 commits into
clone-metadata-copyfrom
hierarchical-classification-pr1

Conversation

@PaulHax

@PaulHax PaulHax commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Add hierarchical track classification

This PR adds an optional type hierarchy to each dataset. For example, a dataset can define fish
as the parent of shark. It can then define shark as the parent of great white shark.

DIVE keeps the existing confidencePairs on each track. The hierarchy is a display and editing
rule. It does not add hierarchy data to tracks or change the annotation schema.

Selection

The viewer selects the deepest checked type that meets its confidence threshold. If great white shark does not meet its threshold, the viewer can select shark.

  • The selected pair controls the displayed type, confidence, and color.
  • Single-camera and multicamera views use the same selection rule.
  • Each camera resolves its own camera track.
  • Lists, counts, and charts use the merged logical-track view.
  • A dataset without a hierarchy keeps the existing flat selection behavior.

Configuration

DIVE stores the hierarchy as typeHierarchy in DIVE Configuration JSON. The value maps each child
type to its direct parent.

  • A hierarchy is a forest in which each child has one parent.
  • DIVE rejects empty names, self-edges, cycles, and conflicting parents.
  • An invalid import does not change the stored configuration.
  • An overwrite import replaces a nonempty hierarchy.
  • null and {} delete the hierarchy during an overwrite import.
  • An additive import adds valid edges. In this mode, null deletes and {} makes no change.
  • Web, Desktop, ZIP export, clone, and multicamera paths carry the configuration.

DIVE Annotation JSON, VIAME CSV, KPF, and NIST files do not carry the hierarchy. PR 5 adds KWCOCO
transport.

Type editing

  • Hierarchy types remain ordinary rows in the Type List. This PR does not add a tree editor.
  • Rename updates all valid hierarchy references and linked camera tracks.
  • DIVE rejects a rename that creates a cycle, a parent conflict, or a duplicate track pair.
  • Delete Type remains blocked while a camera still uses the type.
  • Deleting an unused type removes its filter and style settings. It keeps hierarchy edges.
  • Prevent Cascade Types is disabled while a hierarchy is active. DIVE keeps its saved value.

Counterpart tracks now copy the complete confidence vector from the source track. Delete Type now
checks the tracks in every camera. These two corrections also apply to flat multicamera datasets.

Validation

The server and client use the same validation examples. Server code validates stored hierarchy
data. Client code handles traversal, rename, selection, and editing.

Manual tests

Test data: download and extract
classification.zip. Paths
below are relative to dive-classification-test-data/. If the attachment is not available, generate
the same ZIP from PaulHax/dive-devkit.

Web single-camera import

  1. In Web, create an image-sequence dataset from media/image-sequence/, import
    classification/single-camera-linked-types/tracks.annotations.json and
    type-hierarchy.config.json, then play all frames, save, and reload.

Verify: The three leaf types display, and the hierarchy and confidence pairs remain after reload.

Desktop single-camera import

  1. In Desktop, create an image-sequence dataset from media/image-sequence/, import
    classification/single-camera-linked-types/tracks.annotations.json and
    type-hierarchy.config.json, then play all frames, save, and reload.

Verify: The three leaf types display, and the hierarchy and confidence pairs remain after reload.

Matching multicamera import

  1. Create a multicamera dataset from media/multicamera/, import the matching port and starboard
    files from classification/multicamera-linked-types/, then import its hierarchy configuration.
  2. Switch cameras, play all frames, save, and reload.

Verify: Both cameras show the same three linked tracks and confidence pairs.

Real SEFSC-SEAMAP video import

  1. In Web, create a 5 FPS video dataset from the MP4 in
    classification/sefsc-seamap-fish-taxonomy/, wait for processing, then import its VIAME CSV and
    configuration JSON.
  2. Inspect the tracks and Type List, then save and reload.

Verify: The dataset contains 24 real tracks across eight observed species. The hierarchy includes
seriola_rivolianaseriolacarangidae and remains after reload.

Hierarchy threshold fallback

  1. Create an image-sequence dataset from media/image-sequence/, then import the annotations and
    configuration from hierarchy/valid-three-level-forest/.
  2. Leave all types checked, set bluefin-tuna to 0.95, then set tuna to 0.25.

Verify: The first change selects the parent, and the second change selects the root.

Invalid hierarchy import

  1. Create the valid dataset with
    dive-classification-test-data/hierarchy/valid-three-level-forest/multipair.annotations.json
    and three-level-forest.config.json.
  2. Import one configuration from
    dive-classification-test-data/hierarchy/invalid-configurations/ as DIVE Configuration JSON.

Verify: DIVE rejects the file and keeps the valid hierarchy.

Limits

  • Track attribute filters still use the raw highest-confidence pair in this PR.
  • Bottom-bar assignment still uses the old model mutation in this PR.
  • PR 2 adds canonical logical-track mutation commands.

Stack

This is PR 4 of 9. Previous: isolate soft-clone metadata.
Next: centralize classification changes.

  1. Allow devDependency imports in TypeScript spec files
  2. Preserve warnings from every Desktop import file
  3. Copy source metadata when creating a single-camera soft clone
  4. Current — Add hierarchical track classification
  5. Centralize hierarchical classification changes
  6. Replace mutable merged tracks with read-only projections
  7. Make track lifecycle operations classification-safe
  8. Add lossless DIVE KWCOCO classification support
  9. Define raw and resolved classification boundaries

This PR is stacked on clone-metadata-copy.

Overall: 56 files, +6,213/-385. Commits: 951ad2f1 through f583e248.

Additive configuration import now follows JSON merge semantics at the top
level: an explicit null clears the stored hierarchy, an empty map still makes
no change, and a non-empty map still adds edges.
`markTypeHierarchyPersisted` cleared the dirty flag unconditionally once the
save resolved. A hierarchy edit made while that save was in flight was
acknowledged as persisted and never sent. The acknowledgement now carries the
patch that was actually sent and only clears the flag when the current state
still matches it.

Renaming a type on a flat multicamera dataset walked the merged track view,
whose confidence vector can hide a type an individual camera still carries, so
those tracks kept the old name. `updateTypeName` follows the base pass with a
per-camera pass, next to the equivalent walk its hierarchy branch already does.
`CameraStore.changeTrackTypes` provided this before and has had no callers
since; it goes rather than sit alongside a second implementation.
Python's str.strip() treats U+001C-U+001F and U+0085 as blank and JS's
String.trim() does not; trim() treats U+FEFF as blank and strip() does not. A
name built from those characters was an empty child on one platform and a
legitimate type on the other, so the same configuration validated differently
depending on where it was imported.

Both normalizers test the union instead. The two shared-spec cases cover a
child blank only under strip() and a parent blank only under trim(), so neither
platform can drift again without the other's suite failing.

@BryonLewis BryonLewis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going to approve with some notes about missing or future PRs that should be done.

First internal Notes for me:

  • easily over 2/3 of the changes are tests modifications
  • The front-end touches many files because of a difference in how styling and confidence pairs are passed to UI elements and Layers. This means most UI elements that display types or colors (lots of them) and the Layers that display colors needed to be updated
  • Expansive logic for preventing circular or weird hierarchy data imports
  • care given to supporting multicamera and hierarchy types

Future PRs/Notes:

  • Type list is completely flat even with a hierarchy specified it is set as flat
  • When toggling a parent type I would assume that children should be toggled on/off as well. This can be done after this stack is merged
  • Counts for a parent should be a sum of all of the children types
  • If a confidence pair parent isn't explicitly stated it won't automtically fallover to displaying the parent if the child is toggled off

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants