Repository navigation
docs: reposition EvoCode as a general evolution framework; add attribution CI check - #12
Conversation
The README was titled "VRP Destroy Strategy Evolution" and read as a VRP tool that happened to be extensible, even though the codebase had already been generalized into a domain-plugin architecture with two shipped domains. It had also drifted out of sync with the core/ -> building/ + evaluation/ + domains/ refactor. Repositioning: - Lead as an LLM-guided framework for evolving code components; VRP is now one of two reference domains alongside vm_scheduling - Generalize the architecture diagram (builder -> artifact instead of "compiles to plugin JAR") and add the population layer that was missing: diversity checking, embedding novelty, idea history - Add sections for the Streamlit dashboard, Docker, repository layout, and the /evolve Claude Code skill Accuracy fixes, all verified against current source: - Entry point is `cd evolver && python evo_agent.py`, default config is configs/default.json - Flags: --no-debug (not --no-verbose), plus the undocumented -v/-vv, --log_path, --novisual - `verbose` is an int (0/1/2); `debug` is a separate bool - ABC paths and imports updated for the core/ refactor - Drop num_workers/instance_workers from the config table (constructor-only); max_parallel_evals is the real knob - Drop scripts/run_array.sh, which does not exist - .env belongs in evolver/, and the Java requirement is JDK 21 - Split config into required keys (read via config[...], so a run fails without them) and optional ones - Correct three claims about the evaluator loader, notably that target_instances arrives empty at construction and is assigned by EvolutionLoop afterwards - Fix the ReEvo citation; cite VRPAgent by name only
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4413705f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| | `GEMINI_MODEL` | Model name (default `gemini-2.0-flash-exp`) | | ||
| | `GOOGLE_API_KEY` | Enables the code-embedding service for novelty/visualization (optional) | | ||
| | `MODELARTS_API_KEY`, `MODELARTS_MODEL` | Credentials for the ModelArts / OpenAI-compatible adapter | | ||
| | `EVOCODE_DB_PATH` | SQLite path the dashboard reads (default `/data/evolution.db`) | |
There was a problem hiding this comment.
Make the dashboard honor EVOCODE_DB_PATH
When a user sets EVOCODE_DB_PATH to a writable local or cluster path, the evolution process honors it, but ui/frontend/src/dashboard.py still hardcodes DB_PATH = "/data/evolution.db". The documented dashboard therefore waits indefinitely instead of displaying that run; either read the variable in the dashboard too or document that it affects only the evolution writer.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 4962016.
Verified: ui/frontend/src/dashboard.py:10 hardcodes DB_PATH = "/data/evolution.db", while EVOCODE_DB_PATH is read only by persistence/db_logger.py. So the variable affects the writer alone, and redirecting a run does leave the dashboard empty.
Taking the second of your two options — documenting it — since making the dashboard honor the variable is a behavior change that belongs outside a docs PR. The env table now says "where the evolution run writes its SQLite log", and the Output section carries an explicit warning that the dashboard hardcodes the default path.
Worth a follow-up to have dashboard.py read os.getenv("EVOCODE_DB_PATH", "/data/evolution.db") so the two agree.
| |-----------|---------|-------------| | ||
| | `experiment_name` | `"unnamed"` | Label for the run | | ||
| | `domain` | `"ails_vrp"` | Registered domain plugin name | | ||
| | `source_code_root` | per-domain | Path to the target application (relative to the repo root, or absolute) | |
There was a problem hiding this comment.
Correct the source_code_root resolution base
For both built-in plugins, a relative source_code_root is resolved against the evolver/ directory, not the repository root (AILSVRPPlugin and VmSchedulingPlugin append it to evolver_root). Following this description with "applications/AILS" therefore resolves to the nonexistent evolver/applications/AILS; the shipped configs correctly use "../applications/...".
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct — fixed in 4962016.
Verified in both plugins: evolver_root = Path(__file__).parent.parent.parent.parent resolves to evolver/, and each does (evolver_root / raw).resolve() for a non-absolute value (ails_vrp/plugin.py:37-40, vm_scheduling/plugin.py:42-44). Following my old wording with "applications/AILS" would have produced evolver/applications/AILS, which does not exist.
The table entry now reads: a relative path resolves against evolver/, not the repo root — which is why the shipped configs use "../applications/AILS" — and absolute paths are used as given.
| def get_llm_context(self) -> Dict[str, Any]: return self._builder.get_llm_context() | ||
| def get_initial_seeds(self) -> List[tuple]: return get_initial_seeds() | ||
| def get_instances(self) -> List[str]: return self._evaluator.get_instances() |
There was a problem hiding this comment.
Wire the documented plugin seeds into the builder context
When a new domain follows this recipe, MyDomainPlugin.get_initial_seeds() is never consulted by the evolution loop: EvolutionLoop passes only builder.get_llm_context() to LLMAgents, which reads seeds from that context. Consequently the seeds defined in the shown templates.py and returned here are ignored (and a builder without its own initial_seeds falls back to AILS templates), so the recipe should explicitly expose get_initial_seeds() through the builder context or change the loop to call this hook.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed — and it goes one step further than you flagged. Fixed in 4962016.
grep -rn "get_initial_seeds" evolver/src/ returns only definitions: the ABC, the two plugins, and the templates modules. No call site anywhere. Seeds reach the LLM solely via builder.get_llm_context()["initial_seeds"] — evolution_loop.py:221-222 passes only the builder context to LLMAgents, which does ctx.get("initial_seeds", get_ails_initial_seeds()) at llm_agents.py:162. Both built-in builders populate the key (ails_vrp/builder.py:138, vm_scheduling/builder.py:143), which is why the shipped domains work despite the dead hook.
The same applies to BaseDomainPlugin.get_instances(): instances come from config["target_instances"], falling back to evaluator.get_instances() (evolution_loop.py:195). The plugin's version is never called either.
Took the documentation branch of your suggestion rather than changing the loop, since rewiring it is a behavior change beyond a docs PR. Option C now opens with a callout stating that the builder's context is what feeds the LLM and that a builder omitting the key silently falls back to AILS VRP templates; the plugin example is annotated inline; and the interface reference flags both hooks as unwired.
The cleaner long-term fix is probably to have EvolutionLoop merge plugin.get_initial_seeds() into the context when the builder does not supply it, so the ABC stops advertising a hook that does nothing. Happy to open a separate issue for that.
Follow-ups to the README rewrite, same class of drift: - evo_agent.py: module docstring and argparse --help description said "VRP Destroy Strategy Evolution". The entry point is domain-agnostic; the problem is chosen by the "domain" config key. - requirements.txt: header said "VRP destroy strategy evolution" and claimed JDK 11+. The Java requirement is JDK 21 (what setup.sh and run_evolution.sh load) and applies only to the built-in Java domains. - scripts/README.md: referenced scripts/run_array.sh, which does not exist, and told you to run setup.sh from evolver/ when the script resolves paths from the repo root. Also documented .env in the wrong place, and pointed at a "Python code block" in run_evolution.sh that is actually a config file argument. Adds the SBATCH directive table, and notes that #SBATCH lines are read before the shell runs, so the --account default in run_evolution.sh does not expand $SLURM_ACCOUNT.
Adds a check that fails CI when a commit attributes authorship to an AI
assistant, via the author, the committer, or a Co-authored-by trailer.
The logic lives in scripts/ci/check_attribution.sh so it can be run
locally before pushing:
scripts/ci/check_attribution.sh # commits not on origin/main
scripts/ci/check_attribution.sh main..HEAD
The workflow runs on pull requests and on pushes to main, checking only
the commits the event introduces. Banned patterns are a single ERE at the
top of the script, so other assistants can be added there.
On failure the script prints each offending commit and the exact field
that matched, plus the rebase/amend commands to correct it.
f441370 to
6a6aeca
Compare
All three verified against source: - EVOCODE_DB_PATH affects the evolution writer only. The dashboard hardcodes /data/evolution.db (ui/frontend/src/dashboard.py:10), so redirecting the run leaves the dashboard empty. Documented rather than changed, since making the dashboard read the variable is a behavior change. - A relative source_code_root resolves against evolver/, not the repo root: both plugins do (evolver_root / raw).resolve(). Following the old wording with "applications/AILS" would have pointed at the nonexistent evolver/applications/AILS. - Seeds reach the LLM through builder.get_llm_context()["initial_seeds"], not through the plugin. EvolutionLoop passes only the builder context to LLMAgents, and BaseDomainPlugin.get_initial_seeds() has no call site anywhere in src/ — a builder that omits the key silently falls back to the AILS VRP templates. The same is true of the plugin's get_instances(). Option C now says so, and the interface reference flags both hooks as unwired.
What
README.mdso EvoCode reads as a general LLM-guided framework for evolving code components, rather than a VRP-specific tool. VRP becomes one of two reference domains.evo_agent.py,requirements.txt, andscripts/README.md.Why
The README was titled "VRP Destroy Strategy Evolution" and read as a VRP tool that happened to be extensible — but the codebase already has a domain-plugin architecture with
ails_vrpandvm_schedulingshipped, and a fully domain-agnosticEvolutionLoop. It had also drifted out of sync with thecore/→building/+evaluation/+domains/refactor, so several of its code examples no longer imported correctly.Changes
Positioning
ails_vrp,vm_scheduling); the VRP dataset table moved underneath it/evolveClaude Code skillStale facts corrected
python evo_agent.py,config.jsoncd evolver && python evo_agent.py,configs/default.json--no-verbose--no-debug, plus the undocumented-v/-vv,--log_path,--novisualverbose: trueverboseis an int (0/1/2);debugis a separate boolcore/base_builder.py,from evaluator import …building/base_builder.py,from evaluation import …num_workers/instance_workersconfig keysmax_parallel_evalsis the real knobscripts/run_array.sh.envat repo root, JDK 11+.envinevolver/, JDK 21config[...], so a run crashes without them) and optional onestarget_instancesarrives empty at construction and is assigned toevaluator.target_instancesbyEvolutionLoopafterwards, so a custom evaluator must read the attribute at evaluate time rather than cache it in__init__Beyond the README
evo_agent.py— module docstring and--helpdescription said "VRP Destroy Strategy Evolution"requirements.txt— VRP-specific header; claimed JDK 11+ wheresetup.shandrun_evolution.shboth load JDK 21scripts/README.md— referenced the nonexistentrun_array.sh, told you to runsetup.shfromevolver/when it resolves paths from the repo root, put.envin the wrong place, and pointed at a "Python code block" inrun_evolution.shthat is really a config file argument. Adds an SBATCH directive table and notes that#SBATCHlines are read before the shell runs, so the--accountdefault does not expand$SLURM_ACCOUNT.Attribution CI check
.github/workflows/attribution.ymlfails when a commit attributes authorship to an AI assistant — checking the author, the committer, and anyCo-authored-bytrailer.The logic is in
scripts/ci/check_attribution.shso it can be run locally before pushing:Banned patterns are a single ERE at the top of the script, so other assistants can be added in one place. On failure it prints each offending commit and the exact field that matched, plus the commands to correct it. The workflow runs on pull requests and pushes to
main, checking only the commits the event introduces.Notes for review
evo_agent.pyargparse; loader behavior read fromevaluation/loader.py,building/loader.py, andutils/path_utils.py).dataset_diris documented as defaulting toVrp_Set_X(the fallback inails_vrp/plugin.py), with a note thatconfigs/default.jsonsetsXL.Co-authored-bytrailer on the first commit of this branch, which has since been amended away, and passes on the branch as it stands.One thing this does not cover: the pattern matches the substring
claude, so a human contributor named Claude would be rejected. Worth an allowlist if that ever comes up.