Skip to content

Expect! -> Require!; SAM2 object registration and tracking#274

Open
osnr wants to merge 19 commits into
mainfrom
osnr/sam2-register-tracking
Open

Expect! -> Require!; SAM2 object registration and tracking#274
osnr wants to merge 19 commits into
mainfrom
osnr/sam2-register-tracking

Conversation

@osnr

@osnr osnr commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

I wouldn't say this is a good object tracker API yet (it doesn't undistort; you don't have quads or programs or geometry queries on the contours that it provides; it can only run on one viewport at a time probably), but it works and has some other minor fixes to Folk.

need viewport, registration program, and contour visualizer:

IMG_6876.MOV

Viewport:

Claim $this is a viewport

Registration (needs to be cleared manually):

When the quad library is /ql/ &\
     the image library is /imageLib/ &\
     $this wishes /nobody/ tracks object /anything/ with prompt /anything/ &\
     /v/ is a viewport {
    foreach i {3 2 1} {
       Hold! Wish $this is labelled "Tracking in $i sec..."
       sleep 1
    }
    Hold! {}
    Hold! -key track {}

    Expect! $this has quad /q/
    Expect! $v has quad /vq/

    set prompt [$ql pointToXY $vq [$ql centroid $q]]

    Hold! -key track \
      Wish $v tracks object [list $this object] \
        with prompt $prompt
}

Registration (flip to clear):

When the quad library is /ql/ & /v/ is a viewport {
    foreach i {3 2 1} {
       Hold! Wish $this is labelled "Tracking in $i sec..."
       sleep 1
    }
    Hold! {}
    Hold! -key track {}

    Expect! $this has quad /q/
    Expect! $v has quad /vq/

    set prompt [$ql pointToXY $vq [$ql centroid $q]]

    Hold! -key track \
      Wish $v tracks object [list $this object] \
        with prompt $prompt
}

Contour visualization:

When /obj/ has contours /cs/ in /v/ {

    set color {0 0.8 1 1}
    set thickness 0.0015

    Wish $this is labelled [llength $cs]
    foreach contour $cs {
        Wish $v draws contour $contour \
          with color $color thickness $thickness
    }
}

Put down a viewport (ideally make it large), put a registration page on it, put your object in the center of the registration page, and then it should track and draw the contours.

It can track multiple objects on the viewport (but you need separate registration pages for all the objects you're tracking if you use the design above)

Fixes serialization of the obj id integers in sam2 module and extends it a lot for multi-object prompts and a higher-level track api.

Changes sam2 camera predictor around a lot (I don't fully understand this, but I didn't understand the original predictor code anyway) using some of the newer sam2 changes to their video predictor that let you add objects at runtime, change state around.

Other fixes:

  • Change Expect! to Require!; This matches the traditional meaning of Tcl expect and also communicates that Require! is pretty demanding and is almost an assert, your program will fail if it doesn't match exactly 1.
  • introduce a new Expect! that blocks until a matching statement appears. in some cases, this is interchangeable with Require!, but we really want it for some race-y operations like getting the next camera slice that shows up for a given object, and Expect! feels like the right name
  • Linewrap and downscale titles/other decorations; makes long errors more readable
  • Add contour drawing utility (similar to polyline, but contours have no units. Need to think more about this?)
  • Move quadLib into separate file
    • rename alignGeometry to createFromGeometry (because it is an object constructor basically, so deserves naming as that)
    • add pointToXY that maps from (usually camera-space) xyz point to quad-space xy point. useful for mapping between program coordinate systems, camera, camera slice, drawing!
  • Add imageLib get utility, which lets you probe individual pixels on an image. Basically just the probe utility we used in the marble run and color circle demos

Would be great if you can test normal Folk stuff and make sure there are no regressions or weird breakage of that. Testing SAM2 would be great but not needed. Will merge tomorrow afternoon if no objections

@osnr
osnr requested review from cwervo, ppkn and smj-edison July 22, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant