Sglang Auto Benchmark

• if the benchmark will run from a remote repo copy or ad-hoc synced workspace, verify that the remote python/sglang/bench_serving.py matches the local feature level needed by auto benchmark before launching a long run
• at minimum, run a preflight such as PYTHONPATH=<repo>/python python3 -m sglang.bench_serving --help and confirm that the dataset choices include autobench
• if autobench is missing remotely, do not start the benchmark; sync python/sglang/bench_serving.py and any required dataset modules first

Remote Run Logging

If the benchmark is executed on a remote machine, the progress bar output must be mirrored back to a local file for humans to watch.

Scope note:

• use the remote-log mirroring workflow only when the benchmark is running in a different machine or a different remote container than the one the agent is actively operating in
• if the agent itself is already running inside the target container where auto benchmark is executing, do not add a separate log-return loop just for parity; inspect the live log files and result files directly in the current container
• in other words, "remote container" needs mirrored local logs, while "current container" should use direct local inspection

Required behavior:

• start the remote run with a persistent terminal/session log, for example with script -q -f <log> -c "<cmd>"; on Linux containers that use util-linux script, prefer the explicit -c form instead of BSD-style positional command arguments
• continuously sync a cleaned version of that remote session log back to a local progress.log; this local progress.log should already have terminal control sequences removed, because script + tqdm progress bars will otherwise leave ANSI cursor-control bytes and carriage-return redraws that look like garbled text
• if the benchmark itself is executed inside a remote container, the cleaned local progress.log must be refreshed automatically at least once every 30 seconds while the run is active; do not rely on one-off manual polling
• implement the sync loop as a dedicated local script file checked into neither git nor the benchmark config; avoid fragile one-line nohup zsh -lc '...' command strings with heavy nested quoting
• prefer running the sync loop inside a long-lived local session such as a dedicated tmux pane, screen, or the agent's own persistent PTY session; detached child processes started from short-lived command runners can be reaped unexpectedly, so plain nohup ... & is not the most stable default
• immediately after starting the sync loop, verify that progress.log is actually updating by checking its timestamp or size twice across a short wait; if it is not changing, treat that as a broken sync setup and fix it before telling the user that live log mirroring is working
• tell the user the local log path up front
• keep final result files synced back locally after the run ends
• when scenario-level or top-level markdown summaries are produced, sync those summary.md / SUMMARY.md files back locally as first-class result artifacts rather than leaving them only on the remote machine

This is important because long searches can run for hours, and people need a stable local file they can tail without logging into the remote box. The final local run folder should also be self-contained enough for someone to review the benchmark outcome without re-entering the remote environment.

Recommended cleanup pipeline for the local mirrored log:

perl -pe 's/\e\[[0-9;?]*[ -\/]*[@-~]//g; s/\r/\n/g; s/\x08//g;' raw_progress.log \
  > progress.log

Recommended remote-container sync pattern:

cat > sync_progress.sh <<'EOF'
#!/bin/zsh
set -euo pipefail
while true; do
  ssh <remote-host> "tail -n 200 <remote-progress-log>" > raw_progress.log
  perl -pe 's/\e\[[0-9;?]*[ -\/]*[@-~]//g; s/\r/\n/g; s/\x08//g;' raw_progress.log \
    > progress.log
  sleep 15
done
EOF
chmod +x sync_progress.sh

Run that script from a long-lived local session, for example:

tmux new-session -d -s autobench-sync './sync_progress.sh'

Use a persistent local background job, tmux pane, screen, or equivalent long-lived sync process so that humans can watch the cleaned local log in real time. Use sleep 15 by default for long runs unless there is a specific need for tighter polling, and keep the cleaned local progress.log within the required 30-second refresh window while the run is active.

At the end of the run, make sure the local artifact set includes any generated:

• results.jsonl
• results.csv
• summary.md
• SUMMARY.md
• scenario_summary.jsonl
• scenario_summary.csv

Required health check after starting the sync script:

stat -f '%m %z' progress.log
sleep 5
stat -f '%m %z' progress.log

If the timestamp and size both stay unchanged while the remote benchmark is known to be producing new output, the sync loop is broken. Fix the script before continuing.

Do not make the cleaned log optional. The default local progress artifact should be the cleaned progress.log that humans actually read.

Most Important Rule

If the user wants the best command for a real production or real workload scenario, the benchmark must use their real request distribution.

That means:

• real prompt lengths,
• real output lengths,
• real multi-turn patterns,
• real tool / reasoning / sampling settings,
• real prefix-sharing behavior if it exists.

sharegpt, random, and generated-shared-prefix are useful for sanity checks and broad tuning, but they are not a substitute for the user’s real traffic.

The cookbook reference configs now default to random because it is portable and immediately runnable, but that should still be treated as a fallback benchmark shape rather than the final answer for a real deployment.

Supported Dataset Kinds

The current implementation intentionally keeps the dataset surface small:

• sharegpt
  • Supports auto-download when no file path is provided.
  • Will be prepared into canonical autobench JSONL on disk before benchmarking.
• custom
  • Supports two cases:
    • old bench_serving custom conversation JSONL,
    • already-converted canonical autobench JSONL.
• random
  • Uses SGLang’s existing synthetic/random benchmark path.
  • This is the default dataset mode in the cookbook reference configs.
  • input_len and output_len can be lists of equal length.
  • Each aligned pair becomes one full benchmark scenario, not a cartesian product.
  • Example: