Skip to content

Comments

Fix directional relative selectors to prefer closest element over deepest#16

Open
gdealmeida1885 wants to merge 5 commits intodevicelab-dev:mainfrom
gdealmeida1885:fix/relative-selector-element-selection
Open

Fix directional relative selectors to prefer closest element over deepest#16
gdealmeida1885 wants to merge 5 commits intodevicelab-dev:mainfrom
gdealmeida1885:fix/relative-selector-element-selection

Conversation

@gdealmeida1885
Copy link
Contributor

Summary

Directional relative selectors (below:, above:, leftOf:, rightOf:) incorrectly pick the deepest DOM element instead of the closest by distance. This causes taps to land on the wrong element when a deeply-nested node exists further from the anchor than a shallower, closer match.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

How We Found This Bug

Our app uses an Expo auth WebView (via expo-auth-session) that doesn't expose testID attributes on its elements. To automate the login flow, we rely on relative position selectors to locate input fields — e.g., tapOn: below: "Email Address" to tap the email input.

This worked correctly with standard Maestro, but with maestro-runner, the selector consistently tapped a deeply-nested link far below the anchor instead of the input field directly beneath it.

Real-World Impact

Any screen where elements lack testIDs (WebViews, third-party auth pages, embedded browsers) and relative selectors are used will tap the wrong element when a deeper DOM node exists further from the anchor. This makes auth flows and similar WebView interactions unreliable.

Root Cause Analysis

Maestro vs maestro-runner Comparison

Step Maestro (Kotlin) maestro-runner (Go)
Distance sort Euclidean center-to-center Y-offset only (correct)
Clickable-first sort Stable sort preserving distance order Partition preserving order (correct)
Final selection .firstOrNull() — picks closest DeepestMatchingElement() — picks deepest DOM depth

The pipeline in maestro-runner:

  1. FilterBelow() → returns candidates sorted by distance (closest first) ✅
  2. SortClickableFirst() → moves clickable elements first, preserving order within groups ✅
  3. DeepestMatchingElement()discards distance ordering, picks highest DOM depth ❌

Concrete Example

Page structure:
  "Email Address" label         ← anchor (y=100, height=30, bottom at y=130)
  TextField "email input"       ← y=140, depth 2, distance=10   ✅ should be selected
  <div>
    <div>
      <div>
        Link "email link"       ← y=350, depth 5, distance=220  ❌ was being selected

DeepestMatchingElement picked the Link (depth 5) over the TextField (depth 2), ignoring that the TextField was 22x closer.

Changes Made

Fix (4 locations across 3 drivers)

When a directional filter (below/above/leftOf/rightOf) was applied, use candidates[0] (closest element, clickable-preferred) instead of DeepestMatchingElement(). Non-directional filters (childOf/containsChild/insideOf) continue using depth-based selection since they don't sort by distance.

// Before:
selected = DeepestMatchingElement(candidates)

// After:
if filterType == filterBelow || filterType == filterAbove ||
   filterType == filterLeftOf || filterType == filterRightOf {
    selected = candidates[0]  // distance-sorted, closest wins
} else {
    selected = DeepestMatchingElement(candidates)
}
Driver Function File
WDA (iOS) resolveRelativeSelector pkg/driver/wda/driver.go
Appium findElementRelativeWithElements pkg/driver/appium/driver.go
UIAutomator2 resolveRelativeSelector pkg/driver/uiautomator2/driver.go
UIAutomator2 findElementRelativeWithElements pkg/driver/uiautomator2/driver.go

Tests (3 new test cases)

Each driver gets a regression test that creates a page source with:

  • An anchor label ("Email Address")
  • A close clickable element just below (y=140, shallow depth)
  • A far-but-deeply-nested clickable element further below (y=350, deep depth)

Verifies the closest element is selected, not the deepest.

Testing

  • Tests pass locally (go test ./pkg/driver/...)
  • Added tests for all three drivers (WDA, Appium, UIAutomator2)
  • All existing tests pass with zero regressions
  • Built binary and tested manually on device with auth WebView flow
  • Verified tapOn: below: "Email Address" now taps the input field, not the link

Checklist

  • Code follows project style guidelines
  • Self-reviewed the code
  • No breaking changes
  • Added regression tests for new behavior

Additional Notes

This fix is intentionally scoped to directional filters only. Containment-based filters (childOf, containsChild, insideOf) don't sort by distance, so DeepestMatchingElement remains the correct selection strategy for those.

@omnarayan
Copy link
Contributor

@gdealmeida1885 Thank you, There is conflict, if you can resolve it great else I ll do

CheckCondition() was passing when condition selectors (visible/notVisible)
directly to the driver without expanding variables. Expressions like
${output.homeScreen.buttons.profile} were sent as literal strings,
causing conditions to always evaluate as false and silently skip
conditional blocks.

Three changes:
- CheckCondition(): expand variables in visible/notVisible selectors
  and platform field before evaluating against the driver
- ExpandStep(): add RunFlowStep case to expand variables in File,
  When condition fields, and Env values
- executeNestedStep(): call ExpandStep before executeRunFlow for
  nested RunFlowStep execution
- TestExpandStep_RunFlowStep: verifies ExpandStep expands variables
  in File, When.Visible, When.NotVisible, When.Script, When.Platform,
  and Env map values
- TestExpandStep_RunFlowStep_NilWhen: verifies nil When doesn't panic
- TestCheckCondition_ExpandsVisibleSelectorVariables: verifies expanded
  selector ID reaches the driver
- TestCheckCondition_ExpandsNotVisibleSelectorVariables: same for
  NotVisible text selector
- TestCheckCondition_ExpandsPlatformVariable: verifies platform string
  expansion before comparison
Address PR review feedback: CheckCondition() should only evaluate
conditions, not expand variables. ExpandStep() already handles all
variable expansion before CheckCondition() is called, so the expansion
in CheckCondition() was a redundant second pass.

- Revert CheckCondition() to use condition fields directly
- Remove CheckCondition expansion tests and mockConditionDriver
- Keep ExpandStep RunFlowStep case (the actual fix)
…pest

- Change below/above/leftOf/rightOf selection from DeepestMatchingElement
  to candidates[0] (closest by distance, clickable-preferred)
- Keep deepest element behavior for non-directional filters (childOf, etc.)
- Add regression tests for all three drivers (WDA, Appium, UIAutomator2)
@gdealmeida1885 gdealmeida1885 force-pushed the fix/relative-selector-element-selection branch from f271944 to d40f3ac Compare February 20, 2026 13:38
@gdealmeida1885
Copy link
Contributor Author

@omnarayan I've rebased this onto the latest main and fixed the changelog conflict. Should be good to merge now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants