Android Use

Choosing the Right Device

When the user mentions:

• "phone", "my phone", "physical device" → Use the [PHYSICAL] device
• "emulator", "virtual device", "AVD" → Use the [EMULATOR] device
• If unclear, ask the user which device they want to target

Using the Serial Flag

Once you identify the target device, pass -s <serial> to ALL subsequent scripts:

# Check specific device
scripts/check-device.sh -s 1A051FDF6007PA

# All actions on that device
scripts/get-screen.sh -s 1A051FDF6007PA
scripts/tap.sh -s 1A051FDF6007PA 540 960
scripts/launch-app.sh -s 1A051FDF6007PA chrome

Important: Be consistent - use the same serial for all commands in a session.

Core Workflow

When given a task, follow this perception-action loop:

1. Check device connection - Run scripts/check-device.sh first
   • If multiple devices: identify target based on user intent or ask
   • Note the serial number for subsequent commands
2. Get current screen state - Run scripts/get-screen.sh [-s serial] to dump UI hierarchy
3. Analyze the XML - Read the accessibility tree to understand what's on screen
4. Decide next action - Based on goal + current state, choose an action
5. Execute action - Run the appropriate script with -s serial if needed
6. Wait briefly - Allow UI to update (typically 500ms-1s)
7. Repeat - Go back to step 2 until goal is achieved

Reading UI XML

The get-screen.sh script outputs Android's accessibility XML. Key attributes to look for:

<node index="0" text="Settings" resource-id="com.android.settings:id/title"
      class="android.widget.TextView" content-desc=""
      bounds="[42,234][1038,345]" clickable="true" />

Important attributes:

• text - Visible text on the element
• content-desc - Accessibility description (useful for icons)
• resource-id - Unique identifier for the element
• bounds - Screen coordinates as [left,top][right,bottom]
• clickable - Whether element responds to taps
• scrollable - Whether element can be scrolled
• focused - Whether element has input focus

Calculating tap coordinates: From bounds="[left,top][right,bottom]", calculate center:

• x = (left + right) / 2
• y = (top + bottom) / 2

Example: bounds="[42,234][1038,345]" → tap at x=540, y=289

Available Scripts

All scripts are in the scripts/ directory. Run them via bash.

All scripts support -s <serial> to target a specific device.

Device Management

Screen Reading

Input Actions

App Management

Action Guidelines

When to tap

• Target clickable elements
• Always calculate center from bounds
• Prefer elements with clickable="true"

When to type

• After tapping a text input field
• The field should have focused="true" or class="android.widget.EditText"
• Clear existing text first if needed (select all + delete)

When to swipe

• To scroll lists or pages
• To navigate between screens (e.g., swipe left/right for tabs)
• Directions: up (scroll down), down (scroll up), left, right

When to use keys

• home - Return to home screen
• back - Go back / close dialogs
• enter - Submit forms / confirm
• recent - Open recent apps

When to take screenshots

• For visual debugging when XML doesn't capture enough info
• To verify visual state (colors, images, etc.)
• When the task requires visual confirmation

When to wake the device

• Before starting any task (device may have gone to sleep)
• If get-screen.sh returns empty or minimal XML
• If actions don't seem to be working (screen may be off)
• Note: Won't bypass PIN/pattern/password - user must unlock manually

Common Patterns

Opening an app

# By package name (fastest)
scripts/launch-app.sh com.android.chrome

# By app name (searches installed apps)
scripts/launch-app.sh "Chrome"

Tapping a button

1. Get screen: scripts/get-screen.sh
2. Find element with matching text/content-desc
3. Calculate center from bounds
4. Tap: scripts/tap.sh 540 289

Entering text in a field

1. Tap the text field to focus it
2. Wait for keyboard
3. Type: scripts/type-text.sh "your text here"
4. Press enter if needed: scripts/key.sh enter

Scrolling to find content

1. Get screen to check if target is visible
2. If not found, swipe: scripts/swipe.sh up
3. Get screen again, repeat until found or reached end

Handling dialogs/popups

• Look for elements with text like "OK", "Allow", "Accept", "Cancel"
• Tap the appropriate button
• Or press back to dismiss: scripts/key.sh back

Error Handling

No device connected

• Check USB connection
• Verify USB debugging is enabled
• Run adb devices manually to troubleshoot

Element not found

• The UI may have changed - get fresh screen dump
• Try scrolling to find the element
• Element might be in a different screen/state

Action didn't work

• Wait longer between actions (UI might be slow)
• Verify coordinates are correct
• Check if a popup/dialog appeared

App not responding

• Press home and reopen the app
• Or force close and restart

Example Sessions

Single Device

User request: "Open Chrome and search for weather"

1. scripts/check-device.sh
   → Device connected: Pixel 6
   → Serial: 1A051FDF6007PA
   → Type: Physical

2. scripts/launch-app.sh com.android.chrome
   → Chrome launched

3. scripts/get-screen.sh
   → [Read XML, find search/URL bar]
   → Found: bounds="[0,141][1080,228]" resource-id="com.android.chrome:id/url_bar"
   → Center: x=540, y=184

4. scripts/tap.sh 540 184
   → Tapped URL bar

5. scripts/get-screen.sh
   → [Verify keyboard appeared and field is focused]

6. scripts/type-text.sh "weather"
   → Typed "weather"

7. scripts/key.sh enter
   → Pressed enter to search

8. scripts/get-screen.sh
   → [Verify search results loaded]
   → Task complete!

Multiple Devices

User request: "Open Settings on my phone" (with emulator also running)

1. scripts/check-device.sh
   → Multiple devices connected (2):
   →   [PHYSICAL] 1A051FDF6007PA - Pixel 6
   →   [EMULATOR] emulator-5554 - sdk_gphone64_arm64

   User said "my phone" → target the PHYSICAL device
   Serial to use: 1A051FDF6007PA

2. scripts/check-device.sh -s 1A051FDF6007PA
   → Device connected: Pixel 6
   → Serial: 1A051FDF6007PA
   → Type: Physical
   → Status: Ready

3. scripts/launch-app.sh -s 1A051FDF6007PA settings
   → Resolved 'settings' to package: com.android.settings
   → Launched: com.android.settings

4. scripts/get-screen.sh -s 1A051FDF6007PA
   → [Read XML, verify Settings app is open]
   → Task complete!

Tips

• Be patient - Android UI can be slow, wait between actions
• Read carefully - The XML tells you exactly what's on screen
• Check your work - Get screen after each action to verify state
• Use screenshots - When XML doesn't give enough context
• Start simple - Break complex tasks into small steps
• Multi-device - Always check for multiple devices first; ask user if target is unclear
• Consistent serial - Once you pick a device, use -s <serial> on ALL commands