State graph navigation, workflow recording and replay with tiered verification.
Workflows navigate to a target state using the app's state graph. Verification uses tiered cost escalation: cheapest check first, only escalate if needed.
Level 0: Template Match (~0.3s, 0 tokens)
→ Check target state's defining_components on screen
→ matched_ratio > 0.7 → confirmed ✅
Level 1: detect_all (~2s, 0 tokens)
→ Full component detection + identify_current_state
→ matches expected → continue
→ different known state → re-route via find_path
→ unknown state → escalate to Level 2
Level 2: LLM Fallback
→ Return ("fallback", state, step, reason) for LLM to decide
The agent doesn't know the path. Every click is trial and error:
# Each click records a PENDING transition
click_and_record(app, "Scan", x, y) # pending: unknown → click:Scan
click_and_record(app, "Run", x, y) # pending: click:Scan → click:Run
# Workflow succeeded! Commit all transitions
confirm_transitions(app) # → saved to transitions.json
# OR workflow failed — discard everything
discard_transitions(app) # → nothing saved, graph stays clean
After full end-to-end success:
save_workflow(app, "smart_cleanup", target_state="s_c8e5f3",
description="Navigate to cleanup complete page")
# By name
run_workflow(app, "smart_cleanup")
# By target state directly
execute_workflow(app, "s_c8e5f3")
execute_workflow flow:
find_path(current, target) → BFS shortest pathWorkflows are stored in workflows.json per app (same directory as meta.json):
{
"check_baggage_fee": {
"target_state": "s_c8e5f3",
"description": "Navigate to baggage fee calculator",
"created_at": "2026-03-23 15:30:00",
"last_run": "2026-03-23 16:00:00",
"run_count": 3,
"success_count": 2,
"notes": []
}
}
States are matched using Jaccard similarity against defining_components.
identify_current_state() — pure identification, never creates new statesidentify_or_create_state() — identifies OR creates (for exploration mode)from app_memory import identify_current_state, load_states, load_components
states = load_states(app_dir)
components = load_components(app_dir)
state_id, jaccard = identify_current_state(states, detected_set, components)
Fast verification without full detection:
from app_memory import quick_template_check
matched, total, ratio = quick_template_check(app_dir, ["btn_submit", "nav_bar", "logo"], img=screenshot)
if ratio > 0.7:
print("Target state confirmed")
# Run a workflow
python3 scripts/agent.py run_workflow --app "CleanMyMac X" --workflow smart_cleanup
# View workflows
python3 scripts/agent.py workflows --app "CleanMyMac X"
# View all workflows across all apps
python3 scripts/agent.py all_workflows
# View committed transitions (state graph)
python3 scripts/app_memory.py transitions --app "CleanMyMac X"
# View pending (uncommitted) transitions
python3 scripts/app_memory.py pending --app "CleanMyMac X"
# Commit after success
python3 scripts/app_memory.py commit --app "CleanMyMac X"
# Discard after failure
python3 scripts/app_memory.py discard --app "CleanMyMac X"
# Find path between states
python3 scripts/app_memory.py path --app "CleanMyMac X" --component from_state --contact to_state
execute_workflow() returns one of:
| Result | Meaning |
|---|---|
("success", state_id) | Reached target state |
("fallback", state_id, step_idx, reason) | Need LLM intervention |
("error", message) | Cannot execute (no path, missing state, etc.) |
discard_transitions()confirm_transitions() after full end-to-end success