Bootstrap Realtime Eval

Keep the questions concise and grouped into one short batch whenever possible.

If the user only provides user_text or a short task description, still ask the questions above first. If they answer only partially, then infer the remaining low-risk details, call out the assumptions, and make the scaffold easy to revise later.

Workflow

1. Ask the user for the required inputs first.

   • Do this before making files or selecting a final harness.
   • If the user already supplied some of the inputs, ask only for the missing ones.
   • If you recommend a harness, wait for the user response before scaffolding.

2. Pick the harness.

   • crawl: single-turn text-to-TTS.
   • walk: replay saved audio or generate audio from text rows.
   • run: multi-turn simulation with tool mocks and judge criteria.
   • If the user wants synthetic audio but does not care about replay-specific audio characteristics, prefer crawl over walk.

3. Normalize the inputs.

   • If the user gives inline prompt or tool content, write it into the generated folder.
   • If the user gives CSV data, inspect it with pandas before wiring it in.
   • If the data is not yet harness-ready, scaffold the files and then have you author the starter dataset directly in those files.
   • If the user only gives user_text, infer example_id values and leave optional grading fields blank unless you have enough signal to fill them.
   • Ground the starter data in the use case, prompts, tools, and any other material the user provided.
   • Make the starter data realistic. Put yourself in the shoes of the end user in that use case and craft datapoints that a real user would plausibly say or do.

4. Be proactive when data is missing.

   • crawl: you should author 3 starter rows covering one happy path and a couple of nearby variants or edge cases.
   • walk: you should author 3 source CSV rows and prepare the audio-generation step so the user can create audio immediately.
   • run: you should author 2 starter simulations, not 1.
   • Do not rely on deterministic script-generated samples for use-case-specific starter data.
   • Show the generated starter samples to the user, ask for a quick greenlight or correction, then expand only after feedback when the task calls for more coverage.

5. Run the scaffold script:

python examples/evals/realtime_evals/skills/bootstrap-realtime-eval/scripts/bootstrap_realtime_eval.py --name "<eval_name>" --harness "<crawl|walk|run>"

Add flags for prompt, tools, data, graders, and run-specific fields as needed. Read the script help if you need the exact flag names.

6. Review the generated folder.

   • Confirm the README is accurate for the selected harness.
   • Confirm the system prompt, tools, and data files point at the generated folder.
   • If the user provided real data, make sure it is wired in instead of leaving starter placeholders.
   • If you authored inferred starter data, show the user the generated rows or simulations before scaling up.

7. Enrich the scaffold.

   • For crawl and walk, make the CSV realistic and ensure the expected tool columns are present.
   • For walk, if the dataset lacks audio_path, use the shared walk_harness/generate_audio.py flow described in the README.
   • For run, make sure simulations.csv and the starter sim_*.json file reflect the user’s scenario, tool mocks, and graders.

8. Validate before returning.

   • Run a smoke eval for the generated folder.
   • If the smoke eval succeeds, automatically run the full eval for the generated folder in the same turn.
   • If the smoke eval fails, stop and fix or report the blocker before attempting the full eval.
   • Run pytest examples/evals/realtime_evals/tests -q.
   • Inspect the generated README and at least one generated data file.

Hard Rules

• Keep the generated folder inside examples/evals/realtime_evals/.
• Ask the user for the missing setup information before scaffolding. Do not jump straight to building a default eval when the request is to create a new eval.
• Do not duplicate the main harness scripts into the generated folder.
• Do not write generic placeholder starter data when the use case provides enough context to do better.
• Use the prompt, tools, and scenario details to make the initial datapoints feel like a real user interaction.
• The README must include:
  • why this harness was chosen
  • which files to edit first
  • smoke and full run commands
  • data contract notes
  • troubleshooting notes
• Prefer assistant-specific flags for run harness commands:
  • --assistant-system-prompt-file
  • --assistant-tools-file

Completion Criteria

Treat the task as complete only when:

• the folder exists under examples/evals/realtime_evals/<name>_realtime_eval/
• README.md, system_prompt.txt, and tools.json exist
• harness-specific starter data exists, authored by you when the user did not provide it
• the smoke command has been run or is blocked for a clear reason
• if the smoke command succeeded, the full eval command has also been run or is blocked for a clear reason
• pytest examples/evals/realtime_evals/tests -q has been run

Learnings

When this skill uncovers a reusable workflow or harness constraint that should guide future bootstrap work, add a short note here.

Keep learnings concise and action-oriented:

• Problem -> Fix -> Why

Only add items that are likely to help future realtime-eval scaffolding in this repo. Remove stale items when they no longer apply.

• gpt-realtime temperature is unsupported -> Do not add a temperature field or CLI flag when scaffolding gpt-realtime evals -> Avoids invalid config and keeps runs aligned with the realtime harness constraints.
• Starter samples should come from the model, not the scaffold script -> Use the script to create the folder structure and template files, then have you author the initial rows or simulations from the user’s use case -> Produces more relevant starter data and makes iteration easier with the user.
• Synthetic audio can be overfit to the wrong harness -> Default unspecified synthetic-audio requests to crawl text-to-TTS and reserve walk for replay-specific audio characteristics like noise or telephony artifacts -> Keeps the bootstrap path simpler unless audio realism is the actual target.
• Smoke-only validation can leave scaffolds half-proven -> After a successful smoke run, automatically run the full eval before declaring the bootstrap complete -> Catches dataset-wide or late-row failures that a one-example smoke test misses.