Teleoperation for LeKiwi 3-omni-wheel base. Includes Xbox controller and iOS phone (HEBI app) methods.
Teleoperation for LeKiwi robot. Two methods: iPhone (recommended) or Xbox controller.
Phone is easier. No Bluetooth pairing, no stick drift, natural tilt control.
Use iPhone with HEBI Mobile I/O app to drive base via tilt. Official LeRobot approach.
Install app: https://docs.hebi.us/tools.html#mobile-io Open app → Set family="HEBI", name="mobileIO"
cd ~/projects/lerobot
source .venv/bin/activate
python examples/phone_to_lekiwi/teleoperate.py
| Input | Action |
|---|---|
| Hold B1 | Enable driving |
| Tilt forward/back | Move forward/back |
| Tilt left/right | Strafe left/right |
| Twist phone | Turn left/right |
| A3 slider | Speed multiplier |
# Verify motors detected
ls -la /dev/ttyACM0
# Pair Xbox controller
bluetoothctl scan on
# Press PAIR button on controller (3 sec, fast flash)
bluetoothctl pair <MAC>
bluetoothctl trust <MAC>
bluetoothctl connect <MAC>
# Verify controller visible
ls -la /dev/input/js*
# Put Hermes wrappers on PATH (one-time setup in ~/.bashrc)
case ":$PATH:" in
*":$HOME/.hermes/bin:"*) ;;
*) export PATH="$HOME/.hermes/bin:$PATH" ;;
esac
# Start host first (dry-run for smoke test)
lekiwi-zmq-host --dry-run
# In another terminal, run teleop against that host
lekiwi-teleop --host localhost
# Real hardware host
lekiwi-zmq-host --port /dev/ttyACM0
cd ~/projects/lerobot
source .venv/bin/activate
# Base only
python lekiwi_xbox_base_only.py
# With camera view
python lekiwi_teleop_with_cameras.py --cam 0
# Test controller without robot
python lekiwi_teleop_with_cameras.py --dry-run
| Input | Action |
|---|---|
| Left stick | Move (forward/back/strafe) |
| Right stick X | Rotate left/right |
| RB / LB | Speed up / down (3 levels) |
| A button | Emergency stop |
| Start button | Quit |
--arm) - Direct Joint ControlDirect joint control is recommended over IK. IK causes drift (up 3s then down 3s doesn't return to same position). AI policies use joint control anyway.
| Input | Action |
|---|---|
| Left stick Y | shoulder_lift (UP=extend, DOWN=retract) |
| Left stick X | elbow_flex |
| RB + left stick X | shoulder_pan (LEFT/RIGHT) |
| LB + left stick | wrist_flex / wrist_roll |
| Right stick Y | wrist_flex (alternative) |
| Right trigger | gripper (idle=open, pressed=close) |
| D-pad | Base motion |
| Y / A | Base speed up / down |
| Back | Reset arm + recover stuck servos |
| Start | Toggle left stick arm control (son safety) |
| Ctrl+C | Quit |
Parallel gripper velocity control: Parallel grippers must use velocity mode - position control would over-current fault when gripping objects. Servo stops moving on contact instead of fighting. Trigger mapping:
Velocity defaults: 150 servo units (floor 60 for auto-throttle protection). Positive velocity = open, negative = close. If claw moves wrong direction, swap velocity signs.
Responsive arm control:
Parallel gripper velocity control:
# Tune gripper limits for your parallel gripper
python lekiwi_xbox_arm_teleop.py --gripper-open 85 --gripper-close 35
# Adjust velocity if too fast/slow
python lekiwi_xbox_arm_teleop.py --gripper-vel 150 # slower
python lekiwi_xbox_arm_teleop.py --gripper-vel 400 # faster
Deadzone adjustment: If robot drifts with stick centered, increase deadzone (default 0.15):
python lekiwi_xbox_base_only.py --deadzone 0.2
Speed tuning: Edit speed levels in script for personal preference. Default conservative for indoor use.
XLeRobot uses flat dict protocol - no message wrapping.
Commands IN (port 5555, PULL):
{"x.vel": 0.1, "y.vel": 0.0, "theta.vel": 30.0}
Observations OUT (port 5556, PUSH):
{"x.vel": 0.1, "y.vel": 0.0, "theta.vel": 30.0, "cam_0": "/9j/4AAQ..."}
NOT wrapped format (wrong):
{"type": "response", "response": "video", "data": {"frame": "..."}}
The LeKiwi host sends raw flat dictionaries matching upstream LeRobot's lekiwi_host.py format.
5555 - Commands (PULL socket) - receives flat action dict from web GUI5556 - Data (PUSH socket) - sends flat observation dict with base64 camera frames to web GUI# On Pi 5 - start host
lekiwi-zmq-host
# With cameras
lekiwi-zmq-host --cam 0 --cam 1
# Flip camera vertically (upside-down mount)
lekiwi-zmq-host --cam 0 --camera-vflip
# Debug mode (verbose logging)
lekiwi-zmq-host --debug
Camera vflip:
If camera is mounted upside-down on chassis, use --camera-vflip flag. This flips image vertically in software. The flip includes .copy() to ensure contiguous memory for JPEG encoding (OpenCV flip returns non-contiguous array which breaks cv2.imencode).
The host binds to 0.0.0.0:5555 and 0.0.0.0:5556 so any client can connect.
In your XLeRobot web control, set:
ROBOT_HOST=<pi-ip-address>
ROBOT_PORT_CMD=5555
ROBOT_PORT_DATA=5556
Or if web GUI is on the same Pi:
ROBOT_HOST=localhost
LeKiwi uses Innomaker U20 1080p USB cameras (2x: base + wrist).
Specs:
cv2.VideoCapture(0)Setup:
# Plug in cameras - should auto-detect
ls /dev/video* # Look for new video devices
# Test camera
python ~/projects/lerobot/test_camera.py
# Check supported formats
v4l2-ctl -d /dev/video0 --list-formats-ext 2>/dev/null
Finding Camera Indices:
USB cameras typically show as /dev/video0, /dev/video2, etc.
# In Python - try indices until one works
for i in range(10):
cap = cv2.VideoCapture(i)
if cap.isOpened():
print(f"Camera {i} working")
cap.release()
USB Webcams:
# Find available cameras
cd ~/projects/lerobot
source .venv/bin/activate
python test_camera.py
# List video devices
ls /dev/video*
v4l2-ctl --list-devices
The teleop script with camera can be extended to record episodes for fine-tuning SmolVLA:
# Record episode (drive around, collect data)
python lekiwi_teleop_with_cameras.py --record --output dataset/
# Later: fine-tune SmolVLA on collected data
For now: drive and see camera feed before training.
Document progress with built-in log system:
cd ~/projects/LeKiwi/log
# Add entry with media
python add_entry.py "Arm mounted" "First servo attached" --image ~/Pictures/arm.png
# View log
cat PROJECT_LOG.md
Shoulder pan not working (RB + left stick):
left_y (vertical) instead of left_x (horizontal). Fixed - now RB + left stick LEFT/RIGHT controls shoulder_pan.grep "shoulder_pan (left/right" ~/.hermes/projects/lekiwi-control/src/lekiwi/teleop/xbox.pySon safety - Start button toggle:
Shoulder lift doesn't go all the way back (heavy arm):
# In ArmState.LIMITS - test mechanical range
"shoulder_lift": (-90.0, 120.0), # was (-30.0, 120.0)
if name == "shoulder_lift" and current != clamped:
print(f"[CLAMP] shoulder_lift tried {current:.1f}, clamped to {clamped:.1f}")
Shoulder lift direction:
StopIteration error in motors_bus.sync_write:
None for arm positions when only controlling baseLeKiwi.send_action() crashes when arm keys have None values (iterates over models, gets empty)obs = robot.get_observation()
arm_action = {}
for key in obs:
if key.startswith("arm_") and key.endswith(".pos"):
arm_action[key] = obs[key]
full_action = {**base_action, **arm_action}
Missing dependencies for phone teleop:
If you get ModuleNotFoundError for uvicorn, starlette, python-dotenv, etc.:
cd ~/projects/lerobot
uv pip install uvicorn starlette python-dotenv
hebi-py and teleop packages require these but don't declare them properly.
Camera busy / ioctl errors: Phone teleop may fail if cameras are in use by another process (host, another teleop, etc.)
robot_config = LeKiwiConfig(
port="/dev/ttyACM0",
id="my_lekiwi",
cameras={}, # Empty dict = no cameras
disable_torque_on_disconnect=False,
)
HEBI app not connecting:
hostname -IPhone arm teleop on LeKiwi: what actually worked / failed
Critical fixes if experimenting with examples/phone_to_lekiwi/teleoperate_arm.py:
LeKiwiConfig(id=None) so it loads ~/.cache/huggingface/lerobot/calibration/robots/lekiwi/None.json.robot.connect(calibrate=False), cache calibration in software with:
if robot.calibration:
robot.bus.calibration = robot.calibration
Do not call robot.bus.write_calibration(robot.calibration) on this setup — some homing offsets exceed Feetech sign-magnitude limits and crash.robot.get_observation() returns arm keys like arm_shoulder_pan.pos, not observation.state.arm_*. If you mirror current arm positions, read those exact keys.None feedback (fbk is None) and must be handled without exiting.EEBoundsAndSafety can throw ValueError: EE jump ... on normal phone jitter. Catch it and freeze/hold last action instead of letting the script die.Recommended control strategy on 5V power:
Arm teleop crashes with KeyError: 'arm_shoulder_pan.pos':
LeKiwi.send_action() safety clamp path expects arm present-position keys without .pos, but arm actions include .pos.max_relative_target=None (the current lekiwi_xbox_arm_teleop.py default) so commands bypass that broken clamp path.Arm servos stuck/at limits (overload fault):
robot.bus.disable_torque(), sleeps 100ms, then enable_torque().Systematic Feetech servo debugging: When servos work briefly then stop, or feel weak, check these registers directly:
# Raw bus debug - works when LeRobot classes fail
from scservo_sdk import PortHandler, PacketHandler
port = PortHandler("/dev/ttyACM0")
packet = PacketHandler(1.0)
port.openPort()
port.setBaudRate(1_000_000)
# Read servo ID 2 (shoulder_lift) status
pos, _, err = packet.read2ByteTxRx(port, 2, 56) # Present_Position
load, _, _ = packet.read2ByteTxRx(port, 2, 60) # Present_Load
volt, _, _ = packet.read1ByteTxRx(port, 2, 62) # Present_Voltage
temp, _, _ = packet.read1ByteTxRx(port, 2, 63) # Present_Temperature
status, _, _ = packet.read1ByteTxRx(port, 2, 65) # Status (overload bit)
if load > 32767: load -= 65536 # Sign conversion
overload = "OVERLOAD!" if status & 0x20 else "OK"
print(f"Pos={pos} Load={load} Volt={volt/10:.1f}V Temp={temp}C {overload}")
Key findings:
5V servos vs 7.4V/12V servos: Check servo label - STS3215 come in different voltage variants:
Common issue: Running 7.4V servos at 5V causes weak torque, stalling under heavy arm weight.
Servo stalls when arm extended (shoulder_lift maxes out):
from scservo_sdk import PortHandler, PacketHandler
port = PortHandler("/dev/ttyACM0")
packet = PacketHandler(1.0)
port.openPort()
port.setBaudRate(1_000_000)
# Monitor ID 2 (shoulder_lift) while moving arm
volt, _, _ = packet.read1ByteTxRx(port, 2, 62)
load, _, _ = packet.read2ByteTxRx(port, 2, 60)
if load > 32767: load -= 65536
print(f"Voltage: {volt/10:.1f}V Load: {load:+d}")
# Load approaching ±26000 = maxed out, will fault soon
# Cap at -30 instead of -90 to stay in mechanical advantage zone
"shoulder_lift": (-30.0, 120.0), # was (-90.0, 120.0)
Less extension = less torque needed = works on 5V temporarilyFixes for under-voltage:
degree_step (smaller movements = less current spike)Servos drop torque when script exits (arm collapses):
LeKiwiConfig disables torque on disconnect.disable_torque_on_disconnect=False in config:
robot_config = LeKiwiConfig(
port="/dev/ttyACM0",
disable_torque_on_disconnect=False, # Keep torque on exit
)
robot.disconnect() from finally blocks if you want immediate reconnect capability without re-homing.Shoulder pan stops responding after hitting limit:
for joint, target in target_positions.items():
control = current[joint] + kp * (target - current[joint])
# Clamp to prevent limit crashes
low, high = JOINT_LIMITS[joint]
control = max(low, min(high, control))
action[f"arm_{joint}.pos"] = control
P-control clamping bug (wrong direction/interpolation):
# WRONG - clamps control output, breaks direction
control = current + kp * (target - current)
control = max(low, min(high, control)) # Bad!
Gripper appears stuck closed (or open) at startup despite correct display values:
--dry-run mode (simulation only, no hardware commands sent)grip=85.0) but servos never receive commands# Check what's running
ps aux | grep lekiwi
# Kill dry-run host
pkill -f lekiwi-host
# Start real hardware host
/home/rocky-1/.hermes/projects/lekiwi-control/.venv/bin/lekiwi-host --port /dev/ttyACM0
# In another terminal, start teleop
/home/rocky-1/.hermes/projects/lekiwi-control/.venv/bin/lekiwi-teleop
--dry-run when expecting physical movementGripper appears stuck closed (or open) at startup despite correct teleop display:
lekiwi-host may be running with --dry-run flag (simulation only, no hardware commands sent)grip=85.0) but servos never receive any commandps aux | grep lekiwi-host — if it shows --dry-run, that's your problempkill -f lekiwi-host
/home/rocky-1/.hermes/projects/lekiwi-control/.venv/bin/lekiwi-host --port /dev/ttyACM0
Teleop connects, but host replies with Port is in use! on Goal_Position / Present_Position:
lekiwi-control, the publish loop calls robot.get_observation() while the command thread calls robot.send_action().src/lekiwi/host/__init__.py:
from threading import Lock
...
self.robot_lock = Lock()
...
def publish_observation(self):
with self.robot_lock:
obs = self.robot.get_observation()
...
def apply_action(self, action):
with self.robot_lock:
self.robot.send_action(action)
...
Ghost processes causing "Port is in use" errors in logs after supposed kill:
systemctl kill fails if service isn't actually a systemd servicepkill -9 -f lekiwi leaves defunct PIDs that keep serial port openps aux | grep lekiwi and kill -9 <pid> manually, or just rebootps aux before trusting "fixed"Servo intermittent on bus (ID 1 not found):
lerobot.connect() reports missing ID 1 but raw serial scan finds it, check/reseat cable.Shoulder_lift works briefly then stops (1-2 seconds): Heavy arm weight causes overload. Servo works until protection triggers.
Check with raw bus monitor:
from scservo_sdk import PortHandler, PacketHandler
port = PortHandler("/dev/ttyACM0")
packet = PacketHandler(1.0)
port.openPort()
port.setBaudRate(1_000_000)
# Read servo ID 2 (shoulder_lift)
load, _, _ = packet.read2ByteTxRx(port, 2, 60) # Present_Load
status, _, _ = packet.read1ByteTxRx(port, 2, 65) # Status
if load > 32767: load -= 65536 # Sign conversion
overload = status & 0x20 # Bit 5 = load overload
Look for:
Fix options:
Avoiding overload in first place: Arm servos fault/lock up when hitting limits and don't respond until power cycle:
statuses = robot.bus.sync_read("Status", arm_motors)
for motor, status in statuses.items():
if status & 0x20: # Load overload
print(f"[OVERLOAD] {motor} faulted, recovering...")
robot.bus.write("Torque_Enable", motor, 0)
time.sleep(0.2)
robot.bus.write("Torque_Enable", motor, 1)
# CRITICAL: Use normalize=True to get degrees, not raw encoder counts
pos = robot.bus.read("Present_Position", motor, normalize=True)
self.target_positions[motor.replace("arm_", "")] = float(pos)
normalize=True — raw encoder counts (0-4095) vs degrees causes massive target jump and re-fault.Arm teleop crashes with KeyError: 'arm_shoulder_pan.pos':
LeKiwi.send_action() safety clamp path expects arm present-position keys without .pos, but arm actions include .pos.max_relative_target=None (the current lekiwi_xbox_arm_teleop.py default) so commands bypass that broken clamp path.Arm teleop buttons spam speed/reset/quit while held:
Reset arm or Base speed level every frame.Left stick feels dead or unreliable in IK mode:
IK drift (doesn't return to same position):
Parallel gripper over-current fault:
robot.bus.write("Operating_Mode", "arm_gripper", OperatingMode.VELOCITY.value)
# Left trigger = positive velocity (close)
# Right trigger = negative velocity (open)
# Release = velocity 0 (stop)
Web UI shows "noise" or frozen frame:
{type: "response", data: {...}} wrapping){"cam_0": "base64...", "x.vel": 0.0}Camera vflip not working (corrupted image):
cv2.flip() returns non-contiguous array.copy() after flip: cv2.flip(frame, 0).copy()Port already in use (Address already in use):
# Kill zombies on ports 5555/5556
sudo fuser -k 5555/tcp 5556/tcp
# Then restart host
lekiwi-zmq-host --cam 0
Host crashes immediately with SyntaxError: invalid syntax at type NameOrID = str | int:
lerobot package uses Python 3.12 type-alias syntax (type X = ...).lekiwi-control is running under Python 3.11, host startup will fail before hardware connect.~/.hermes/projects/lekiwi-control/.venv/bin/python --version
lerobot checkout compatible with Python 3.11.pyproject.toml saying requires-python = ">=3.11" is misleading if the bundled lerobot dependency needs 3.12 syntax.Controller not detected:
bluetoothctl info <MAC> # Check Connected: yes
# If not connected:
bluetoothctl connect <MAC>
Motors not responding:
# Check motor IDs
.venv/bin/python -c "
from lerobot.motors.feetech import FeetechMotorsBus
bus = FeetechMotorsBus('/dev/ttyACM0', {})
bus.connect()
print('Found:', bus.motors)
"
Why base-only approach:
LeKiwiBaseOnly minimal class that only talks to wheel motors (IDs 7,8,9)Kinematics:
If the user's ~/.hermes/projects/lekiwi-control teleop is misbehaving but ~/projects/lerobot/lekiwi_xbox_teleop.py is confirmed working on hardware, do not try to outsmart it.
Use the working lerobot script as the source of truth:
~/projects/lerobot/lekiwi_xbox_teleop.py over ~/.hermes/projects/lekiwi-control/src/lekiwi/teleop/xbox.py exactlylekiwi-control/src/lekiwi/teleop/__init__.py compatible by aliasing TeleopConfig = XboxTeleopConfigRecommended parity checks:
cd ~/.hermes/projects/lekiwi-control
uv run pytest -q tests/test_xbox_teleop_parity.py tests/test_xbox_teleop_bdd.py
cmp -s src/lekiwi/teleop/xbox.py ~/projects/lerobot/lekiwi_xbox_teleop.py && echo EXACT || echo DIFFERENT
uv run lekiwi-teleop --help
Recommended regression files:
tests/test_xbox_teleop_parity.py — byte-for-byte parity against upstream scripttests/test_xbox_teleop_bdd.py + tests/bdd/features/xbox_teleop_parity.feature — critical-path Gherkin parity specThis is one of those rare times where duplication is correct. Working beats elegant.
If ~/projects/lerobot/lekiwi_xbox_teleop.py is the version that actually works on hardware, stop being clever and mirror it exactly into ~/.hermes/projects/lekiwi-control/src/lekiwi/teleop/xbox.py.
Rules:
lekiwi-teleop pointing at lekiwi.teleop.xbox:main.TeleopConfig but upstream renamed it to XboxTeleopConfig, fix the local package wrapper or tests around it — not the mirrored teleop file.tests/test_xbox_teleop_parity.py@smokeuv run pytest -quv run pytest -q -m smokeuv run python -m py_compile src/lekiwi/teleop/xbox.py src/lekiwi/teleop/__init__.pyuv run lekiwi-teleop --helpWhat bit us:
teleop_mod.TeleopConfig, but the mirrored upstream file exposed XboxTeleopConfig.TeleopConfig = XboxTeleopConfig in src/lekiwi/teleop/__init__.py and adjust the export test.~/.hermes/projects/lekiwi-control/.venv/bin/lekiwi-teleop~/.hermes/projects/lekiwi-control/.venv/bin/lekiwi-host~/.hermes/bin/~/projects/lerobot/~/projects/LeKiwi/log/PROJECT_LOG.md