Shinka Run

Workflow

1. Inspect task directory

ls -la <task_dir>

Confirm evaluate.py and initial.<ext> exist.

2. Inspect CLI reference quickly

shinka_run --help

3. Check model availability before proposing a run

shinka_models
shinka_models --verbose

Validate the exact run config against shinka_models:

• Mutation models: every entry in evo.llm_models must appear in the llm list.
• Meta recommendation models: if evo.meta_rec_interval is set and evo.meta_llm_models is set, every meta model must appear in the llm list.
• Prompt evolution models: if evo.evolve_prompts=true, use evo.prompt_llm_models when provided, otherwise evo.llm_models; every selected model must appear in the llm list.
• Embedding model: if evo.embedding_model is set, it must appear in the embedding list.
• Local OpenAI-compatible models are allowed for LLMs and embeddings via local/<model>@http(s)://host[:port]/v1, and these local models are not expected to appear in shinka_models.

Important runtime rules:

• Do not assume meta recommendations fall back to evo.llm_models. In the current runner, meta recommendations are only enabled when evo.meta_llm_models is explicitly set.
• Prompt evolution does fall back to evo.llm_models when evo.prompt_llm_models is unset.
• Treat local/<model>@http(s)://host[:port]/v1 values as an explicit exception to the shinka_models membership check. Instead, confirm the local endpoint URL and serving status separately before running.
• If any required model is missing from shinka_models, stop and ask the user to either change the config or set the missing credentials first.

4. Confirm first-batch configuration with the user

• Minimum: budget scope, generation count, critical overrides.
• Explicitly confirm the mutation LLMs, meta recommendation LLMs, prompt evolution LLMs, and embedding model after checking them against shinka_models.
• If unclear, ask before running.
• Do not override any non-confirmed arguments.

5. Launch main run with explicit knobs

shinka_run \
  --task-dir <task_dir> \
  --results_dir <results_dir> \
  --num_generations 40 \
  --set db.num_islands=3 \
  --set job.time=00:10:00 \
  --set evo.task_sys_msg='<task-specific system message guiding search>'\
  --set evo.llm_models='["gpt-5-mini","gpt-5-nano"]' \
  --set evo.meta_llm_models='["gpt-5-mini"]' \
  --set evo.prompt_llm_models='["gpt-5-mini"]' \
  --set evo.embedding_model='text-embedding-3-small' \
  # Concurrency settings for parallel sampling and evaluation
  --max-evaluation-jobs 2 \
  --max-proposal-jobs 2 \
  --max-db-workers 2

6. Verify outputs before handoff

ls -la <results_dir>

Expect artifacts like run log, generation folders, and SQLite DBs.

7. Between-batch handoff (unless explicitly autonomous)

• Summarize outcomes from the finished batch.
• Ask user for the next batch config before running again.
• Explicitly ask: "What new directions should we push next batch? Please include algorithm ideas, constraints, and failure modes to avoid."
• Turn user feedback into a revised system prompt and pass it via --set evo.task_sys_msg=... in the next shinka_run call.
• If the prompt is long/multiline, put it in a config file and use --config-fname instead of shell-escaping.
• Unless the user explicitly wants a fresh run/fork, keep the same --results_dir for follow-up batches.

Example next-batch command with feedback-driven prompt:

shinka_run \
  --task-dir <task_dir> \
  --results_dir <results_dir> \
  --num_generations 20 \
  --set evo.task_sys_msg='<new system prompt derived from user feedback>' \
  --set db.num_islands=3

Batch Control Policy (Required)

Treat one shinka_run invocation as one batch of program evaluations/generations.

• Default mode: human-in-the-loop between batches.
• After each batch and before the first, always ask the user what configuration to run next (budget, --num_generations, model/settings overrides, concurrency, islands, output path).
• Do not start the next batch until the user confirms the next config.
• Keep --results_dir fixed across continuation batches so Shinka can reload prior results.
• Exception: if the user explicitly asks for fully autonomous execution, you may continue across batches without re-asking between runs.