Macpilot Ui Inspector

Find Elements by Text

macpilot ui find "Save" --app "TextEdit" --json           # Find by label
macpilot ui find "Save" --role AXButton --json            # Filter by role
macpilot ui find "Save" --exact --json                    # Exact match only

Find Element with Coordinates

macpilot ui find-text "Submit" --app "Safari" --json
# Returns: position {x, y}, size {w, h} - use for clicking

Click UI Element by Label

macpilot ui click "Save" --app "TextEdit" --json
macpilot ui click "Cancel" --role AXButton --json

Get/Set Values

macpilot ui get-value "Search" --app "Safari" --json          # Read field value
macpilot ui set-value "Search" "query text" --app "Safari"    # Set field value
macpilot ui set-value "Dark Mode" "1" --role AXCheckBox       # Toggle checkbox

Set Focus

macpilot ui set-focus "Search" --app "Safari" --json    # Focus an element

Scroll Within Elements

macpilot ui scroll "content" down 5 --app "Safari"     # Scroll element

Inspect Element Attributes

macpilot ui attributes "Save" --app "TextEdit" --json   # All AX attributes

Elements at Coordinates

macpilot ui elements-at 500 300 --json                  # What's at x=500 y=300
macpilot ui elements-at 500 300 --radius 50 --json      # Search wider area

Accessibility Tree

macpilot ui tree --app "Finder" --json                  # Full AX tree
macpilot ui tree --app "Finder" --depth 3 --json        # Limit depth

Keyboard Shortcuts

macpilot ui shortcuts --app "Safari" --json             # All shortcuts
macpilot ui shortcuts --app "Safari" --menu File --json # Menu-specific

Wait for Elements

macpilot wait element "Download Complete" --app "Safari" --timeout 30 --json

Common AX Roles

Workflow Patterns

Pattern 1: Find and Click

# Always focus app first
macpilot app focus "Safari"
# Find the element to verify it exists
macpilot ui find "Downloads" --app "Safari" --role AXButton --json
# Click it
macpilot ui click "Downloads" --app "Safari"

Pattern 2: Read Form State

macpilot app focus "System Settings"
macpilot ui get-value "Computer Name" --app "System Settings" --json

Pattern 3: Fill a Form

macpilot app focus "MyApp"
macpilot ui set-value "Name" "John Doe" --app "MyApp"
macpilot ui set-value "Email" "[email protected]" --app "MyApp"
macpilot ui click "Submit" --app "MyApp"

Pattern 4: Explore Unknown UI

# Start with a broad scan
macpilot ui tree --app "SomeApp" --depth 2 --json
# Narrow down to specific elements
macpilot ui list --app "SomeApp" --role AXButton --json
# Inspect a specific element
macpilot ui attributes "Settings" --app "SomeApp" --json

Pattern 5: Click by Coordinates (Fallback)

# When label-based clicking fails, find coordinates first
macpilot ui find-text "Submit" --app "MyApp" --json
# Use returned x,y to click
macpilot click 450 320

Tips

• Always call macpilot app focus before any UI interaction
• Use --json output for reliable parsing of element properties
• When ui click fails, fall back to ui find-text + macpilot click x y
• Use ui tree --depth 2 first to understand app structure before deep scans
• AX permission must be granted to MacPilot.app in System Settings
• The --role filter significantly speeds up element searches in complex apps