Interactive Xcuitest

2. Create an Interactive Control Test

Create a UI test in your project's UI test target:

import XCTest
import XCUITestControl

final class InteractiveControlTests: XCTestCase {
    @MainActor
    func testInteractiveControl() throws {
        let app = XCUIApplication()
        app.launch()
        InteractiveControlLoop().run(app: app)
    }
}

3. Get the CLI

Clone the xcode-sim-automation repo to get the CLI tool:

git clone https://github.com/gestrich/xcode-sim-automation.git

The CLI is the wrapper script at Tools/xcuitest-control within the cloned repo. It auto-builds the Swift CLI binary on first run and whenever source files change — no manual build step needed.

A Python fallback (Tools/xcuitest-control.py) is also available if the Swift toolchain isn't installed.

CLI

The Tools/xcuitest-control script provides a simple interface for controlling XCUITest. It accepts the same flags as the Python CLI:

# Tap a button
Tools/xcuitest-control tap --target submitButton --target-type button

# Scroll down
Tools/xcuitest-control scroll --direction down

# Type text
Tools/xcuitest-control type --value "Hello World"

# Adjust a slider to 75%
Tools/xcuitest-control adjust --target volumeSlider --value 0.75

# Pinch to zoom in (scale > 1.0)
Tools/xcuitest-control pinch --scale 2.0 --target imageView

# Wait 2 seconds
Tools/xcuitest-control wait --value 2.0

# Take screenshot
Tools/xcuitest-control screenshot

# Check status
Tools/xcuitest-control status

# Exit the test
Tools/xcuitest-control done

CLI Output

Each command returns JSON with paths to the latest hierarchy and screenshot:

{
  "status": "completed",
  "hierarchy": "/tmp/xcuitest-hierarchy.txt",
  "screenshot": "/tmp/xcuitest-screenshot.png"
}

On error:

{
  "status": "error",
  "error": "Element 'missingButton' not found after waiting 10 seconds",
  "hierarchy": "/tmp/xcuitest-hierarchy.txt",
  "screenshot": "/tmp/xcuitest-screenshot.png"
}

Workflow

1. Start the XCUITest

Run the interactive control test using xcodebuild:

xcodebuild test \
  -workspace YourApp.xcworkspace \
  -scheme "YourUITestScheme" \
  -destination 'platform=iOS Simulator,name=iPhone 16 Pro' \
  -only-testing:"YourUITestTarget/InteractiveControlTests/testInteractiveControl" &

Replace YourApp.xcworkspace, YourUITestScheme, and YourUITestTarget with your project's values.

The test will:

• Launch the app
• Write initial hierarchy and screenshot
• Begin polling for commands

2. Wait for Test Initialization

Poll until the hierarchy file exists:

while [ ! -f /tmp/xcuitest-hierarchy.txt ]; do sleep 1; done

Or use the status command:

Tools/xcuitest-control status

3. Execute Commands

Use the CLI to execute actions:

# Read current UI state
cat /tmp/xcuitest-hierarchy.txt

# View screenshot
# Read file: /tmp/xcuitest-screenshot.png

# Execute action
Tools/xcuitest-control tap --target settingsButton --target-type button

# View updated screenshot after action
# Read file: /tmp/xcuitest-screenshot.png

4. Exit Gracefully

When the goal is achieved:

Tools/xcuitest-control done

CLI Commands Reference

tap

Taps an element by identifier.

Tools/xcuitest-control tap --target submitButton --target-type button
Tools/xcuitest-control tap -t submitButton -T button
Tools/xcuitest-control tap --target Edit --target-type button --index 0

Options:

• --target, -t (required): Accessibility identifier of the element
• --target-type, -T (optional): Element type - button, staticText, cell, textField, slider, or any
• --index, -i (optional): 0-based index when multiple elements match. If omitted, taps the first hittable element.

scroll

Scrolls content in a direction (reveals content in that direction).

Important: The direction specifies where you want to scroll TO (what content to reveal), not the swipe gesture direction:

• --direction down = reveal content below (internally swipes up)
• --direction up = reveal content above (internally swipes down)
• --direction left = reveal content to the left (internally swipes right)
• --direction right = reveal content to the right (internally swipes left)

Tools/xcuitest-control scroll --direction down   # Scroll down to see more content below
Tools/xcuitest-control scroll -d up --target scrollView  # Scroll up to see content above

Options:

• --direction, -d (required): up, down, left, or right - the direction to scroll content
• --target, -t (optional): Element to scroll. If omitted, scrolls the app.

type

Types text into a text field.

Tools/xcuitest-control type --value "[email protected]"
Tools/xcuitest-control type -V "Hello" --target usernameField

Options:

• --value, -V (required): Text to type
• --target, -t (optional): Text field to type into. If omitted, types into currently focused field.

adjust

Adjusts a slider to a normalized position (0.0 to 1.0).

Tools/xcuitest-control adjust --target volumeSlider --value 0.75
Tools/xcuitest-control adjust -t volumeSlider -V 0.5

Options:

• --target, -t (required): Accessibility identifier of the slider
• --value, -V (required): Normalized position between 0.0 (minimum) and 1.0 (maximum)

Examples:

• --value 0.0 - Move slider to minimum (left)
• --value 0.5 - Move slider to middle
• --value 1.0 - Move slider to maximum (right)

pinch

Pinches to zoom in or out on an element.

Tools/xcuitest-control pinch --scale 2.0 --target imageView
Tools/xcuitest-control pinch -s 0.5 -V 2.0

Options:

• --scale, -s (required): Scale factor
  • < 1.0 = pinch in (zoom out)
  • > 1.0 = pinch out (zoom in)
• --velocity, -V (optional): Speed in scale factor per second (default: 1.0)
• --target, -t (optional): Element to pinch. If omitted, pinches the app.

Examples:

• --scale 2.0 - Zoom in 2x
• --scale 0.5 - Zoom out to 50%
• --scale 1.5 --velocity 0.5 - Slow zoom in

wait

Pauses for a specified duration.

Tools/xcuitest-control wait --value 2.0
Tools/xcuitest-control wait  # defaults to 1.0 second

Options:

• --value, -V (optional): Seconds to wait. Defaults to 1.0.

screenshot

Captures current state without performing any action.

Tools/xcuitest-control screenshot

status

Checks current command status without executing.

Tools/xcuitest-control status

done

Exits the test loop.

Tools/xcuitest-control done

Handling Multiple Matches

When multiple elements share the same identifier (e.g., multiple "Edit" buttons in a list), the tap command:

1. Without --index: Automatically finds and taps the first hittable element
2. With --index N: Taps the element at the specified 0-based index

Success Response with Multiple Matches

When a tap succeeds on one of multiple matches, the response includes info:

{
  "status": "completed",
  "info": "Tapped button at index 0 of 5 matches",
  "hierarchy": "/tmp/xcuitest-hierarchy.txt",
  "screenshot": "/tmp/xcuitest-screenshot.png"
}

Error Response for Ambiguous Elements

When multiple elements match but none are hittable:

{
  "status": "error",
  "error": "Found 5 elements matching 'Edit', none were hittable. Specify --index 0 to 4 to select a specific element.",
  "hierarchy": "/tmp/xcuitest-hierarchy.txt",
  "screenshot": "/tmp/xcuitest-screenshot.png"
}

Index Out of Range

When the specified index exceeds available matches:

{
  "status": "error",
  "error": "Index 10 out of range. Found 5 'Edit' element(s). Use --index 0 to 4.",
  "hierarchy": "/tmp/xcuitest-hierarchy.txt",
  "screenshot": "/tmp/xcuitest-screenshot.png"
}

Best Practices

1. Check the hierarchy first to see how many matching elements exist
2. Use --index when you know which element you want (e.g., the second Edit button)
3. Let the framework auto-select when you want any visible/hittable match
4. Review the info field to verify which element was tapped

Keyboard Handling

When interacting with text fields, the keyboard will appear and may block other UI elements.

Dismissing the Keyboard

Tap on a non-interactive element that's visible above the keyboard:

Tools/xcuitest-control tap --target notesLabel --target-type staticText

Tips for dismissing the keyboard:

• Look in the hierarchy for StaticText elements (labels) that are above the keyboard
• Navigation bar titles work well as tap targets
• Section headers or form labels are good choices
• Avoid tapping on text fields, text views, or other interactive elements

Typing Text

1. Tap the text field first to focus it:

   Tools/xcuitest-control tap --target searchBar --target-type any
   

2. Then type your text:

   Tools/xcuitest-control type --value "Hello"
   

Common Keyboard Issues

Reading the UI Hierarchy

The hierarchy file shows the element tree with types, identifiers, and labels:

Application, pid: 12345, label: 'MyApp'
  Window, 0x600000001234
    Other, identifier: 'mainView'
      Button, identifier: 'settingsButton', label: 'Settings'
      StaticText, identifier: 'welcomeLabel', label: 'Welcome!'
      Cell, identifier: 'item_1', label: 'First Item'
      Slider, identifier: 'volumeSlider', value: '50%'

From this hierarchy:

• settingsButton is a Button → --target-type button
• welcomeLabel is a StaticText → --target-type staticText
• item_1 is a Cell → --target-type cell
• volumeSlider is a Slider → --target-type slider

Use --target-type any if unsure - it searches all element types.

Error Handling and Recovery

Robustness Configuration

The skill enforces several limits to prevent runaway behavior:

Command Errors

On error, the CLI returns:

{
  "status": "error",
  "error": "Element 'missingButton' not found after waiting 10 seconds",
  "hierarchy": "/tmp/xcuitest-hierarchy.txt",
  "screenshot": "/tmp/xcuitest-screenshot.png"
}

When this happens:

1. Read the hierarchy to find the correct element
2. Try alternative identifiers or element types
3. Consider if navigation went to an unexpected view
4. Check if the element needs to be scrolled into view

Common Errors

Example Session

Goal: Navigate to the Settings view and explore

# 1. Start the test
xcodebuild test \
  -workspace YourApp.xcworkspace \
  -scheme "YourUITestScheme" \
  -destination 'platform=iOS Simulator,name=iPhone 16 Pro' \
  -only-testing:"YourUITestTarget/InteractiveControlTests/testInteractiveControl" &

# 2. Wait for initialization
while [ ! -f /tmp/xcuitest-hierarchy.txt ]; do sleep 1; done

# 3. Read initial state
cat /tmp/xcuitest-hierarchy.txt

# 4. Read the screenshot to see the current view
# Use the Read tool on /tmp/xcuitest-screenshot.png

# 5. Tap an element based on what you see
Tools/xcuitest-control tap --target someButton --target-type button

# 6. Read updated hierarchy and screenshot
cat /tmp/xcuitest-hierarchy.txt

# 7. Exit when done
Tools/xcuitest-control done

Environment Variable Overrides

The CLI supports environment variable overrides for file paths:

When using custom paths, also configure the Swift InteractiveControlLoop.Configuration to match:

let config = InteractiveControlLoop.Configuration(
    commandPath: "/custom/path/command.json",
    hierarchyPath: "/custom/path/hierarchy.txt",
    screenshotPath: "/custom/path/screenshot.png"
)
InteractiveControlLoop(configuration: config).run(app: app)

File-Based Protocol (Advanced)

For direct JSON manipulation, the CLI uses these files:

Command JSON Schema

{
  "action": "tap" | "scroll" | "type" | "wait" | "screenshot" | "adjust" | "pinch" | "done",
  "target": "elementIdentifier",
  "targetType": "button" | "staticText" | "cell" | "textField" | "slider" | "any",
  "index": 0,
  "value": "text to type (for type) or 0.0-1.0 (for adjust)",
  "direction": "up" | "down" | "left" | "right",
  "scale": "pinch scale factor (< 1.0 = zoom out, > 1.0 = zoom in)",
  "velocity": "pinch speed in scale factor per second",
  "status": "pending" | "executing" | "completed" | "error",
  "errorMessage": "optional error description",
  "info": "optional diagnostic info (e.g., which index was tapped)"
}

Tips for Effective Control

1. Always read hierarchy first - Don't guess element identifiers
2. Use specific target-type - Faster and more reliable than any
3. Handle errors gracefully - Read hierarchy after errors to adapt
4. Wait after animations - Use the wait command if UI is animating
5. Take screenshots often - Helps verify you're on the expected view
6. Exit cleanly - Always run done command when finished
7. Track action count - Monitor progress against the 100 action limit
8. Handle keyboard - Dismiss by tapping non-interactive labels
9. Retry with alternatives - Use --target-type any if specific type fails