Understand and explore the Omi desktop macOS app's UI flows, navigation patterns, and SwiftUI architecture. Use when developing features, fixing bugs, or verifying changes in desktop/ Swift files. Provides agent-swift commands to explore the live app, understand how screens connect, and verify your work.
This skill teaches you the Omi desktop macOS app's navigation structure, screen architecture, and SwiftUI patterns. Use it when developing features (to understand how the app works), fixing bugs (to navigate to the affected screen), or verifying changes (to confirm your code works in the live app).
You can interact with the running app via agent-swift — a CLI that clicks elements, reads the accessibility tree, and captures screenshots through the macOS Accessibility API. Works with any macOS app, no app-side instrumentation needed.
# App must be running via ./run.sh from desktop/
agent-swift doctor # check Accessibility permission
agent-swift connect --bundle-id com.omi.desktop-dev # connect to Omi Dev
agent-swift snapshot -i --json # see what's on screen
| Command | Purpose | Example |
|---|---|---|
snapshot -i --json | See all interactive elements with refs, types, labels | agent-swift snapshot -i --json |
click @ref | CGEvent click — SwiftUI elements (NavigationLink, gestures) | agent-swift click @e3 |
press @ref | AXPress — AppKit buttons, Settings sidebar items | agent-swift press @e5 |
find role/text/key VALUE | Find element and chain action | agent-swift find text "Settings" click |
fill @ref "text" | Type into text field | agent-swift fill @e7 "search" |
scroll down/up | Scroll current view | agent-swift scroll down |
wait text "X" | Wait for element to appear | agent-swift wait text "Loading" --timeout 5000 |
is exists @ref | Assert element exists (exit 0/1) | agent-swift is exists @e3 |
get PROP @ref | Read property value | agent-swift get value @e5 --json |
screenshot PATH | Capture app window | agent-swift screenshot /tmp/screen.png |
Key rules:
click = CGEvent mouse click (SwiftUI). Use for main sidebar icons, NavigationLink.press = AXPress action (AppKit). Use for Settings sidebar sections.find with chained action is more stable than hardcoded @ref numbers.--json flag on any command gives structured output for parsing.Main Window
├── Sidebar (SidebarView.swift) — use `click`
│ ├── Home (DesktopHomeView.swift)
│ ├── Conversation (ChatSessionsSidebar.swift)
│ ├── brain → Memories
│ ├── checklist → Action Items
│ ├── puzzlepiece.fill → Integrations
│ └── gearshape.fill → Settings
│
└── Settings (SettingsPage.swift) — use `press` for sidebar sections
├── General — app preferences
├── Rewind — screenshot/timeline settings
├── Transcription — Language Mode (Auto-Detect / Single Language)
│ └── Language picker (popupbutton or button)
├── Notifications — alert preferences
├── Privacy — data settings
├── Account — user info
├── AI Chat — chat model settings
├── Advanced — developer options
└── About — version info
System Tray Menu
├── openOmi — Open Omi
├── checkFor — Check for Updates
├── resetOnb — Reset Onboarding
├── reportIs — Report Issue
├── signOut — Sign Out
└── quitApp — Quit
Main sidebar navigation:
image type elements with accessibility identifiers: sidebar_dashboard, sidebar_chat, sidebar_memories, sidebar_tasks, sidebar_rewind, sidebar_apps, sidebar_settingsfind key sidebar_dashboard click for reliable navigation (survives UI changes)click — these are SwiftUI views with onTapGestureSettings sidebar navigation:
button type elements with section name labelspress — these are SwiftUI Button views that respond to AXPressTranscription language mode:
click on the text to switch modespopupbutton)menuitem elementsSystem tray menu:
identifier prefixes for detectionsnapshot --json (includes menu bar items)Reference flows in desktop/e2e/flows/*.yaml describe the app's key user journeys. Read these to understand navigation paths, expected elements, and UI state at each step.
| Flow | Covers | Steps | Report |
|---|---|---|---|
flows/navigation.yaml | SidebarView, DesktopHomeView | 6/6 PASS | report |
flows/dashboard.yaml | DashboardPage, GoalsWidget, TasksWidget | 3/6 (3 skipped) | report |
flows/chat.yaml | ChatPage, ChatProvider | 5/5 PASS | report |
flows/memories.yaml | MemoriesPage, MemoryGraphPage | 5/6 (1 skipped) | report |
flows/tasks.yaml | TasksPage, TasksStore | 4/5 (1 skipped) | report |
flows/settings.yaml | SettingsPage, SettingsSidebar | 9/9 PASS | report |
flows/language.yaml | SettingsPage, SettingsSidebar | 5 steps | — |
flows/rewind.yaml | RewindPage | 4/4 PASS | report |
flows/apps.yaml | IntegrationsPage | 6/6 PASS | report |
flows/refer.yaml | ReferPage | 3/3 PASS | report |
flows/screen-recording-permission.yaml | RewindPage, ScreenCaptureService, PermissionsPage | 7/7 PASS | report |
flows/audio-recording.yaml | ConversationsPage, AudioCaptureService, AppState | 7/7 PASS | report |
When you modify a Swift file, check if any flow's covers: includes it. That flow describes the user journey your change affects.
Create desktop/e2e/flows/<name>.yaml in v2 format: