Argent Simulator Interact

3. Opening Apps

Never navigate to an app by tapping home-screen icons. Use launch-app or open-url — they are instant and reliable.

launch-app — by bundle ID

{ "udid": "<UDID>", "bundleId": "com.apple.MobileSMS" }

Common IDs: com.apple.MobileSMS (Messages), com.apple.mobilesafari (Safari), com.apple.Preferences (Settings), com.apple.Maps, com.apple.Photos, com.apple.mobilemail, com.apple.mobilenotes, com.apple.MobileAddressBook (Contacts)

open-url — by URL scheme

{ "udid": "<UDID>", "url": "messages://" }

Common schemes: messages://, settings://, maps://?q=<query>, tel://<number>, mailto:<address>, https://... (Safari)

4. Choosing the Right Tool

5. Finding Tap Targets

IMPORTANT. When moved to a different screen after an action or do not know the coordinates of component, always perform proper discovery first.

Point follow-up native diagnostics after you already have a candidate point:

• native-user-interactable-view-at-point: deepest native view that would receive touch at a known raw iOS point; requires bundleId
• native-view-at-point: deepest visible native view at a known raw iOS point; requires bundleId

If describe Fails

Read the exact error and choose the action that matches it:

• Error mentions ax-service not available or daemon startup failure: the ax-service daemon could not start. Check that the simulator is booted. Use screenshot as a temporary fallback, or use native-describe-screen with an explicit bundleId if the app has native devtools injected.
• describe returns an empty element list: the screen may be blank, loading, or showing content without accessibility labels. Use screenshot to see what is visible, then retry after the content has loaded.
• describe succeeds but is not detailed enough for a React Native app: use debugger-component-tree next.
• You need app-scoped inspection with full UIKit properties (accessibilityIdentifier, viewClassName): use native-describe-screen with an explicit bundleId. This requires native devtools (dylib) injection — call restart-app first if needed.
• You already have a candidate point and want to confirm what would actually receive touch: use native-user-interactable-view-at-point. Use native-view-at-point when you want the visually deepest view instead of the hit-test target.

6. Tool Usage

gesture-tap — Single tap at a point

{ "udid": "<UDID>", "x": 0.5, "y": 0.5 }

Coordinates: 0.0 = left/top, 1.0 = right/bottom.

Before tapping near the bottom of the screen in React Native apps, check that "Open Debugger to View Warnings" banners are not visible — tapping them breaks the debugger connection. Close them with the X icon if present.

gesture-swipe — Straight-line gesture

{ "udid": "<UDID>", "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 }

Swipe up (fromY > toY) = scroll content down. Default duration: 300ms. Optional: "durationMs": 500 for slower swipe.

gesture-pinch — Two-finger pinch

{ "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.2, "endDistance": 0.6 }

All values are normalized 0.0–1.0 (fractions of screen, not pixels) — same as all other gesture tools. startDistance: 0.2 means fingers start 20% of the screen apart; endDistance: 0.6 means they end 60% apart. startDistance < endDistance = pinch out (zoom in). startDistance > endDistance = pinch in (zoom out). Defaults: angle: 0 (horizontal), durationMs: 300. Optional: "angle": 90 for vertical axis, "durationMs": 500 for slower pinch.

gesture-rotate — Two-finger rotation

{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "radius": 0.15,
  "startAngle": 0,
  "endAngle": 90
}

All positions and radius are normalized 0.0–1.0 (fractions of screen, not pixels). radius: 0.15 means each finger is 15% of the screen away from center. endAngle > startAngle = clockwise. Default duration: 300ms. Optional: "durationMs": 500 for slower rotation.

gesture-custom — Custom touch sequence

For long-press, drag-and-drop, and other complex sequences, see references/gesture-examples.md. Set "interpolate": 10 to auto-generate smooth intermediate Move events between keyframes.

button — Hardware button press

{ "udid": "<UDID>", "button": "home" }

Values: home, back, power, volumeUp, volumeDown, appSwitch, actionButton

paste — Type text into focused field

{ "udid": "<UDID>", "text": "Hello, world!" }

Tap the field first, then paste. Fall back to keyboard if it doesn't work.

keyboard — Type text or press special keys

{ "udid": "<UDID>", "text": "search query", "key": "enter" }

Special keys: enter, escape, backspace, tab, space, arrow-up, arrow-down, arrow-left, arrow-right, f1–f12. Optional: "delayMs": 100 between keystrokes (default 50ms).

rotate — Change orientation

{ "udid": "<UDID>", "orientation": "LandscapeLeft" }

Values: Portrait, LandscapeLeft, LandscapeRight, PortraitUpsideDown

7. Screenshots

Use the explicit screenshot tool only when:

• You need the initial screen state before any action.
• The auto-attached screenshot shows a transitional or loading frame.
• You require extra context.
• You want to check state after a delay (e.g. waiting for a network response).
• A permission dialog, system alert, or native modal overlay is visible and describe did not expose reliable targets.

When using screenshot for permission or native modal navigation:

• Do not switch to screenshot-driven navigation just because a modal is visible. On regular app screens and in-app modals, keep using describe.
• Prefer obvious, centered alert buttons such as Allow, OK, Don't Allow, Not Now, or Continue.
• Tap one control at a time and inspect the returned auto-screenshot before doing anything else.
• After the modal is dismissed, return to normal discovery with describe, native-describe-screen, or debugger-component-tree.

Optional rotation parameter: { "udid": "<UDID>", "rotation": "LandscapeLeft" } — rotates the capture without changing simulator orientation.

Screenshots are downscaled by default (30% of original resolution) to reduce context size. scale accepts values from 0.01 to 1.0. If UI elements are hard to read or you need to inspect fine detail, pass scale: 1.0 to get full resolution: { "udid": "<UDID>", "scale": 1.0 }.

Troubleshooting

8. Action Sequencing with run-sequence

Use run-sequence to batch multiple interaction steps into a single tool call. Only one screenshot is returned — after all steps complete. Use cases: scrolling multiple times, typing and submitting automatically, known sequence of multiple taps, rotating device back and forth.

Do not use run-sequence when any step depends on observing the result of a previous step

Use cases

Use the sequencing when:

• Knowing that some action needs multiple steps without necessarily immediate insight of screenshot
• "scroll to bottom", "scroll to top", "scroll to do X" -> sequence scroll 3-5 times
• form interactions, "clear and retype field" -> you may use triple-tap to select all, type new value
• "submit form" → fill all fields in sequence, tap submit
• "go back to X" → defined tap sequence for the navigation

Allowed tools inside run-sequence

gesture-tap, gesture-swipe, gesture-custom, gesture-pinch, gesture-rotate, button, keyboard, rotate

The udid is shared — do not include it in each step's args. Optional delayMs per step (default 100ms).

Examples

Scroll down three times:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } },
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } },
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } }
  ]
}

Type into a focused field and submit:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "keyboard", "args": { "text": "hello world" } },
    { "tool": "keyboard", "args": { "key": "enter" } }
  ]
}

Tap a known button, then scroll down:

{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.15 } },
    {
      "tool": "gesture-swipe",
      "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 },
      "delayMs": 300
    }
  ]
}

Stops on the first error and returns partial results.