Mooncake Deployment Troubleshooting

The metadata server is critical for node discovery and coordination.

# Test etcd connectivity
curl -s http://127.0.0.1:2379/version

# Or test custom metadata server
curl -s $MC_METADATA_SERVER

# Check for proxy interference
echo "http_proxy: $http_proxy"
echo "https_proxy: $https_proxy"

Common issues:

• Error from etcd client → Metadata server unreachable
  • Fix: Ensure etcd is bound to 0.0.0.0 not 127.0.0.1:

    etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379
    

  • Fix: Disable HTTP proxy:

    unset http_proxy https_proxy
    

3. Environment Variables Check

Verify critical environment variables are set correctly:

# Display all MC_* variables
env | grep ^MC_

# Key variables to check:
echo "MC_METADATA_SERVER: $MC_METADATA_SERVER"
echo "MC_FORCE_TCP: $MC_FORCE_TCP"
echo "MC_LOG_LEVEL: $MC_LOG_LEVEL"
echo "MC_YLT_LOG_LEVEL: $MC_YLT_LOG_LEVEL"
echo "MC_MS_AUTO_DISC: $MC_MS_AUTO_DISC"
echo "MC_MS_FILTERS: $MC_MS_FILTERS"
echo "MC_GID_INDEX: $MC_GID_INDEX"
echo "MC_MTU: $MC_MTU"
echo "MC_IB_PORT: $MC_IB_PORT"
echo "MC_ENABLE_DEST_DEVICE_AFFINITY: $MC_ENABLE_DEST_DEVICE_AFFINITY"

Key variables:

• MC_METADATA_SERVER - Metadata server URL (required)
• MC_FORCE_TCP=true - Force TCP for testing without RDMA
• MC_LOG_LEVEL=0 - Enable verbose logging (0=INFO, 1=WARNING, 2=ERROR)
• MC_YLT_LOG_LEVEL=debug - yalantinglibs log level
• MC_MS_AUTO_DISC=1 - Enable topology auto-discovery (default)
• MC_MS_FILTERS - Filter specific RDMA devices (e.g., "mlx5_1,mlx5_2")
• MC_GID_INDEX - RDMA GID index (set if GID is all zeros)
• MC_MTU - RDMA MTU size
• MC_ENABLE_DEST_DEVICE_AFFINITY=1 - Reduce QP creation (fix "Failed to create QP")

4. RDMA Device Check

Only run if RDMA is being used (skip if MC_FORCE_TCP=true):

# List RDMA devices
ibv_devices

# Check device details and status
ibv_devinfo

# Check for ACTIVE ports
ibv_devinfo | grep -A 10 "state:"

# Check GID addresses (should NOT be all zeros)
ibv_devinfo | grep -A 20 "GID"

# Check peer memory modules
lsmod | grep peer_mem
lsmod | grep nvidia_peer_mem

# Check QP count (if "Failed to create QP" error)
rdma resource show qp

Common issues:

• No matched device found → RDMA device name in config doesn't exist
  • Fix: Use ibv_devices to get correct device names
• Device XXX port not active → RDMA port not in ACTIVE state
  • Fix: Check cable connections, verify with ibv_devinfo | grep state
  • Fix: Try different port with MC_IB_PORT environment variable
• GID all zeros → Wrong GID index
  • Fix: Set MC_GID_INDEX=1 (or 2, 3 depending on network)
• Failed to create QP: Cannot allocate memory → Too many QPs created
  • Fix: Set MC_ENABLE_DEST_DEVICE_AFFINITY=1

5. Memory and Resource Limits

Check system limits that affect RDMA memory registration:

# Check ulimits
ulimit -a

# Focus on max locked memory
ulimit -l

# Check RDMA device memory limits
ibv_devinfo -v | grep max_mr_size

# Check dmesg for memory errors
dmesg -T | tail -50 | grep -i "out of mr size"

Common issues:

• Failed to register memory: Input/output error → Memory registration limit exceeded
  • Diagnostic: Check max_mr_size with ibv_devinfo -v
  • Fix: Reduce memory allocation or split into smaller chunks
• Cannot allocate memory → ulimit restriction
  • Fix: Set unlimited locked memory:

    ulimit -l unlimited
    

  • Permanent fix: Add to /etc/security/limits.conf:

    * soft memlock unlimited
    * hard memlock unlimited
    

6. Network Connectivity

Test connectivity between nodes:

# Test basic RDMA connectivity
ib_write_bw -d <device_name> -R

# On peer node:
ib_write_bw -d <device_name> -R <server_ip>

# Test GPU Direct RDMA (if CUDA enabled)
ib_write_bw -d <device_name> -R -x gdr

# On peer node:
ib_write_bw -d <device_name> -R -x gdr <server_ip>

# Test DNS resolution
nslookup <connectable_name>
ping <connectable_name>

Common issues:

• connection refused → Incorrect connectable_name or rpc_port
  • Fix: Ensure connectable_name is NOT loopback (127.0.0.1/localhost)
  • Fix: Use actual LAN/WAN IP or valid hostname
• Failed to exchange handshake → RDMA connection setup failure
  • Fix: Verify MTU matches: set MC_MTU environment variable
  • Fix: Verify GID is valid (not all zeros)
  • Fix: Test with ib_send_bw between nodes first

7. Log Analysis

Search logs for common error patterns and their meanings:

Metadata/Connectivity Errors:

• Error from etcd client → Cannot connect to metadata server
• ERR_METADATA → Metadata server communication failed
• ERR_DNS → Invalid local_server_name (not valid DNS/IP)

RDMA Errors:

• No matched device found → RDMA device name doesn't exist
• Device XXX port not active → RDMA port not in ACTIVE state
• Failed to exchange handshake description → RDMA handshake failed
• Failed to modify QP to RTR, check mtu, gid, peer lid, peer qp num → MTU/GID mismatch
• Failed to register memory → Memory registration limit exceeded
• Failed to create QP → Too many QPs, enable MC_ENABLE_DEST_DEVICE_AFFINITY=1
• Worker: Process failed for slice → Network instability
• work request flushed error → Cascading error (find first error)

Store Errors:

• NO_AVAILABLE_HANDLE (-200) → Memory pool exhausted
  • Fix: Increase global_segment_size in setup
  • Fix: Check eviction is working (look for eviction logs)
• LEASE_EXPIRED (-707) → Lease expired during transfer
  • Fix: Increase default_kv_lease_ttl in master startup
• OBJECT_NOT_FOUND (-704) → Object doesn't exist
• SEGMENT_NOT_FOUND (-101) → No available segments
• Failed to get description of XXX → Segment name mismatch
  • Fix: Ensure segment name matches local_hostname from peer

Port/Service Errors:

• bind address already in use → Port conflict
  • Fix: Use different port: --rpc_port=50052

8. Configuration Validation

Verify configuration is correct:

# Check connectable_name is not loopback
hostname -I

# Verify master startup flags
ps aux | grep mooncake_master

# Check if using correct protocol
env | grep MC_FORCE_TCP

Critical checks:

• connectable_name must be non-loopback IP or valid hostname
• MTU and GID configurations must match network environment
• RDMA device names must exist on the machine
• Ports must not be in use by other services

Error Code Quick Reference

Transfer Engine Error Codes

Store Error Codes

Output Format

Provide a structured diagnostic report:

🔍 MOONCAKE DEPLOYMENT DIAGNOSTICS
==================================

✅ PASSED CHECKS:
- Service status: mooncake_master running on port 50051
- Metadata server: etcd accessible at http://127.0.0.1:2379
- Environment: MC_METADATA_SERVER set correctly
- [other passing checks]

❌ FAILED CHECKS:
- RDMA device: mlx5_0 port not ACTIVE (state: PORT_DOWN)
- Memory limits: max locked memory is 64KB (too low)
- [other failures with specific error messages]

⚠️  WARNINGS:
- GID index not set, may cause connection issues
- HTTP proxy variables set, may interfere with metadata server
- [other potential issues]

🔧 RECOMMENDED FIXES:

1. Fix RDMA port status:
   - Check physical cable connections
   - Verify driver configuration
   - Command: ibv_devinfo | grep -A 10 "state:"

2. Increase memory limits:
   ulimit -l unlimited
   # Or permanently in /etc/security/limits.conf:
   * soft memlock unlimited
   * hard memlock unlimited

3. Set GID index:
   export MC_GID_INDEX=1

4. Disable HTTP proxy:
   unset http_proxy https_proxy

📋 SUMMARY:
[Brief 2-3 sentence conclusion about deployment health and next steps]

Troubleshooting Workflow

1. Start simple: Check services and basic connectivity first
2. Read logs carefully: First error is usually root cause (subsequent errors cascade)
3. Test incrementally: Use MC_FORCE_TCP=true to isolate RDMA issues
4. Verify basics: Check connectable_name, ports, env vars before deep diving
5. Use diagnostic tools: ibv_devices, ibv_devinfo, ib_write_bw, curl
6. Reference documentation: Check error codes and troubleshooting guide

Quick Fix Commands

Start metadata server properly:

etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://<your_ip>:2379

Enable verbose logging:

export MC_LOG_LEVEL=0
export MC_YLT_LOG_LEVEL=debug

Force TCP mode for testing:

export MC_FORCE_TCP=true

Fix memory limits:

ulimit -l unlimited

Fix too many QPs:

export MC_ENABLE_DEST_DEVICE_AFFINITY=1

Fix GID issues:

export MC_GID_INDEX=1  # or 2, 3 depending on network

Use different port:

mooncake_master --rpc_port=50052

Now execute the diagnostic checks systematically and provide the structured report.