Skip to content

docs: reposition EvoCode as a general evolution framework; add attribution CI check - #12

Merged
mahdims merged 4 commits into
mainfrom
docs/generalize-readme
Sep 18, 2026
Merged

mahdims merged 4 commits into
mainfrom
docs/generalize-readme

Conversation

@mahdims

@mahdims mahdims commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

What

  1. Rewrites README.md so EvoCode reads as a general LLM-guided framework for evolving code components, rather than a VRP-specific tool. VRP becomes one of two reference domains.
  2. Clears the same drift from evo_agent.py, requirements.txt, and scripts/README.md.
  3. Adds a CI check that rejects AI assistants as commit authors or co-authors.

Why

The README was titled "VRP Destroy Strategy Evolution" and read as a VRP tool that happened to be extensible — but the codebase already has a domain-plugin architecture with ails_vrp and vm_scheduling shipped, and a fully domain-agnostic EvolutionLoop. It had also drifted out of sync with the core/ → building/ + evaluation/ + domains/ refactor, so several of its code examples no longer imported correctly.

Changes

Positioning

  • New lead: EvoCode evolves any component of an existing program — LLM as variation operator, your evaluator as fitness signal, domain-agnostic loop in between
  • Generalized the architecture diagram ("new Java strategy / compiles to plugin JAR" → "new candidate source / Builder → artifact") and added the population layer that was missing entirely: diversity checking, embedding novelty, idea history
  • New Built-in domains table (ails_vrp, vm_scheduling); the VRP dataset table moved underneath it
  • New sections: Streamlit dashboard, Docker, repository layout, and the /evolve Claude Code skill

Stale facts corrected

Was Now
python evo_agent.py, config.json cd evolver && python evo_agent.py, configs/default.json
--no-verbose --no-debug, plus the undocumented -v/-vv, --log_path, --novisual
verbose: true verbose is an int (0/1/2); debug is a separate bool
core/base_builder.py, from evaluator import … building/base_builder.py, from evaluation import …
num_workers / instance_workers config keys removed — constructor-only; max_parallel_evals is the real knob
scripts/run_array.sh removed — does not exist
.env at repo root, JDK 11+ .env in evolver/, JDK 21
ReEvo/VRPAgent citations swapped ReEvo cited correctly; VRPAgent by name only
  • Config table split into required keys (read via config[...], so a run crashes without them) and optional ones
  • Corrected three claims about the evaluator loader carried over from the old text — notably that target_instances arrives empty at construction and is assigned to evaluator.target_instances by EvolutionLoop afterwards, so a custom evaluator must read the attribute at evaluate time rather than cache it in __init__

Beyond the README

  • evo_agent.py — module docstring and --help description said "VRP Destroy Strategy Evolution"
  • requirements.txt — VRP-specific header; claimed JDK 11+ where setup.sh and run_evolution.sh both load JDK 21
  • scripts/README.md — referenced the nonexistent run_array.sh, told you to run setup.sh from evolver/ when it resolves paths from the repo root, put .env in the wrong place, and pointed at a "Python code block" in run_evolution.sh that is really a config file argument. Adds an SBATCH directive table and notes that #SBATCH lines are read before the shell runs, so the --account default does not expand $SLURM_ACCOUNT.

Attribution CI check

.github/workflows/attribution.yml fails when a commit attributes authorship to an AI assistant — checking the author, the committer, and any Co-authored-by trailer.

The logic is in scripts/ci/check_attribution.sh so it can be run locally before pushing:

scripts/ci/check_attribution.sh              # commits not on origin/main
scripts/ci/check_attribution.sh main..HEAD   # explicit range

Banned patterns are a single ERE at the top of the script, so other assistants can be added in one place. On failure it prints each offending commit and the exact field that matched, plus the commands to correct it. The workflow runs on pull requests and pushes to main, checking only the commits the event introduces.

Notes for review

  • Docs plus CI config — no application code changes.
  • Every linked path, CLI flag, config key, and import in the new README was verified against current source (link existence checked programmatically; flags cross-checked against evo_agent.py argparse; loader behavior read from evaluation/loader.py, building/loader.py, and utils/path_utils.py).
  • dataset_dir is documented as defaulting to Vrp_Set_X (the fallback in ails_vrp/plugin.py), with a note that configs/default.json sets XL.
  • The check was tested in both directions — it caught a Co-authored-by trailer on the first commit of this branch, which has since been amended away, and passes on the branch as it stands.

One thing this does not cover: the pattern matches the substring claude, so a human contributor named Claude would be rejected. Worth an allowlist if that ever comes up.

The README was titled "VRP Destroy Strategy Evolution" and read as a VRP
tool that happened to be extensible, even though the codebase had already
been generalized into a domain-plugin architecture with two shipped
domains. It had also drifted out of sync with the core/ -> building/ +
evaluation/ + domains/ refactor.

Repositioning:
- Lead as an LLM-guided framework for evolving code components; VRP is now
  one of two reference domains alongside vm_scheduling
- Generalize the architecture diagram (builder -> artifact instead of
  "compiles to plugin JAR") and add the population layer that was missing:
  diversity checking, embedding novelty, idea history
- Add sections for the Streamlit dashboard, Docker, repository layout, and
  the /evolve Claude Code skill

Accuracy fixes, all verified against current source:
- Entry point is `cd evolver && python evo_agent.py`, default config is
  configs/default.json
- Flags: --no-debug (not --no-verbose), plus the undocumented -v/-vv,
  --log_path, --novisual
- `verbose` is an int (0/1/2); `debug` is a separate bool
- ABC paths and imports updated for the core/ refactor
- Drop num_workers/instance_workers from the config table (constructor-only);
  max_parallel_evals is the real knob
- Drop scripts/run_array.sh, which does not exist
- .env belongs in evolver/, and the Java requirement is JDK 21
- Split config into required keys (read via config[...], so a run fails
  without them) and optional ones
- Correct three claims about the evaluator loader, notably that
  target_instances arrives empty at construction and is assigned by
  EvolutionLoop afterwards
- Fix the ReEvo citation; cite VRPAgent by name only

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4413705f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread README.md Outdated
| `GEMINI_MODEL` | Model name (default `gemini-2.0-flash-exp`) |
| `GOOGLE_API_KEY` | Enables the code-embedding service for novelty/visualization (optional) |
| `MODELARTS_API_KEY`, `MODELARTS_MODEL` | Credentials for the ModelArts / OpenAI-compatible adapter |
| `EVOCODE_DB_PATH` | SQLite path the dashboard reads (default `/data/evolution.db`) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make the dashboard honor EVOCODE_DB_PATH

When a user sets EVOCODE_DB_PATH to a writable local or cluster path, the evolution process honors it, but ui/frontend/src/dashboard.py still hardcodes DB_PATH = "/data/evolution.db". The documented dashboard therefore waits indefinitely instead of displaying that run; either read the variable in the dashboard too or document that it affects only the evolution writer.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 4962016.

Verified: ui/frontend/src/dashboard.py:10 hardcodes DB_PATH = "/data/evolution.db", while EVOCODE_DB_PATH is read only by persistence/db_logger.py. So the variable affects the writer alone, and redirecting a run does leave the dashboard empty.

Taking the second of your two options — documenting it — since making the dashboard honor the variable is a behavior change that belongs outside a docs PR. The env table now says "where the evolution run writes its SQLite log", and the Output section carries an explicit warning that the dashboard hardcodes the default path.

Worth a follow-up to have dashboard.py read os.getenv("EVOCODE_DB_PATH", "/data/evolution.db") so the two agree.

Comment thread README.md Outdated
|-----------|---------|-------------|
| `experiment_name` | `"unnamed"` | Label for the run |
| `domain` | `"ails_vrp"` | Registered domain plugin name |
| `source_code_root` | per-domain | Path to the target application (relative to the repo root, or absolute) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Correct the source_code_root resolution base

For both built-in plugins, a relative source_code_root is resolved against the evolver/ directory, not the repository root (AILSVRPPlugin and VmSchedulingPlugin append it to evolver_root). Following this description with "applications/AILS" therefore resolves to the nonexistent evolver/applications/AILS; the shipped configs correctly use "../applications/...".

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct — fixed in 4962016.

Verified in both plugins: evolver_root = Path(__file__).parent.parent.parent.parent resolves to evolver/, and each does (evolver_root / raw).resolve() for a non-absolute value (ails_vrp/plugin.py:37-40, vm_scheduling/plugin.py:42-44). Following my old wording with "applications/AILS" would have produced evolver/applications/AILS, which does not exist.

The table entry now reads: a relative path resolves against evolver/, not the repo root — which is why the shipped configs use "../applications/AILS" — and absolute paths are used as given.

Comment thread README.md
Comment on lines +448 to +450
def get_llm_context(self) -> Dict[str, Any]: return self._builder.get_llm_context()
def get_initial_seeds(self) -> List[tuple]: return get_initial_seeds()
def get_instances(self) -> List[str]: return self._evaluator.get_instances()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wire the documented plugin seeds into the builder context

When a new domain follows this recipe, MyDomainPlugin.get_initial_seeds() is never consulted by the evolution loop: EvolutionLoop passes only builder.get_llm_context() to LLMAgents, which reads seeds from that context. Consequently the seeds defined in the shown templates.py and returned here are ignored (and a builder without its own initial_seeds falls back to AILS templates), so the recipe should explicitly expose get_initial_seeds() through the builder context or change the loop to call this hook.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — and it goes one step further than you flagged. Fixed in 4962016.

grep -rn "get_initial_seeds" evolver/src/ returns only definitions: the ABC, the two plugins, and the templates modules. No call site anywhere. Seeds reach the LLM solely via builder.get_llm_context()["initial_seeds"] — evolution_loop.py:221-222 passes only the builder context to LLMAgents, which does ctx.get("initial_seeds", get_ails_initial_seeds()) at llm_agents.py:162. Both built-in builders populate the key (ails_vrp/builder.py:138, vm_scheduling/builder.py:143), which is why the shipped domains work despite the dead hook.

The same applies to BaseDomainPlugin.get_instances(): instances come from config["target_instances"], falling back to evaluator.get_instances() (evolution_loop.py:195). The plugin's version is never called either.

Took the documentation branch of your suggestion rather than changing the loop, since rewiring it is a behavior change beyond a docs PR. Option C now opens with a callout stating that the builder's context is what feeds the LLM and that a builder omitting the key silently falls back to AILS VRP templates; the plugin example is annotated inline; and the interface reference flags both hooks as unwired.

The cleaner long-term fix is probably to have EvolutionLoop merge plugin.get_initial_seeds() into the context when the builder does not supply it, so the ABC stops advertising a hook that does nothing. Happy to open a separate issue for that.

Follow-ups to the README rewrite, same class of drift:

- evo_agent.py: module docstring and argparse --help description said
  "VRP Destroy Strategy Evolution". The entry point is domain-agnostic;
  the problem is chosen by the "domain" config key.
- requirements.txt: header said "VRP destroy strategy evolution" and
  claimed JDK 11+. The Java requirement is JDK 21 (what setup.sh and
  run_evolution.sh load) and applies only to the built-in Java domains.
- scripts/README.md: referenced scripts/run_array.sh, which does not
  exist, and told you to run setup.sh from evolver/ when the script
  resolves paths from the repo root. Also documented .env in the wrong
  place, and pointed at a "Python code block" in run_evolution.sh that
  is actually a config file argument.

Adds the SBATCH directive table, and notes that #SBATCH lines are read
before the shell runs, so the --account default in run_evolution.sh
does not expand $SLURM_ACCOUNT.
Adds a check that fails CI when a commit attributes authorship to an AI
assistant, via the author, the committer, or a Co-authored-by trailer.

The logic lives in scripts/ci/check_attribution.sh so it can be run
locally before pushing:

    scripts/ci/check_attribution.sh              # commits not on origin/main
    scripts/ci/check_attribution.sh main..HEAD

The workflow runs on pull requests and on pushes to main, checking only
the commits the event introduces. Banned patterns are a single ERE at the
top of the script, so other assistants can be added there.

On failure the script prints each offending commit and the exact field
that matched, plus the rebase/amend commands to correct it.
@mahdims
mahdims force-pushed the docs/generalize-readme branch from f441370 to 6a6aeca Compare September 18, 2026 16:56
@mahdims mahdims changed the title docs: reposition README as a general evolution framework docs: reposition EvoCode as a general evolution framework; add attribution CI check Sep 18, 2026
All three verified against source:

- EVOCODE_DB_PATH affects the evolution writer only. The dashboard
  hardcodes /data/evolution.db (ui/frontend/src/dashboard.py:10), so
  redirecting the run leaves the dashboard empty. Documented rather
  than changed, since making the dashboard read the variable is a
  behavior change.

- A relative source_code_root resolves against evolver/, not the repo
  root: both plugins do (evolver_root / raw).resolve(). Following the
  old wording with "applications/AILS" would have pointed at the
  nonexistent evolver/applications/AILS.

- Seeds reach the LLM through builder.get_llm_context()["initial_seeds"],
  not through the plugin. EvolutionLoop passes only the builder context
  to LLMAgents, and BaseDomainPlugin.get_initial_seeds() has no call
  site anywhere in src/ — a builder that omits the key silently falls
  back to the AILS VRP templates. The same is true of the plugin's
  get_instances(). Option C now says so, and the interface reference
  flags both hooks as unwired.
@mahdims
mahdims merged commit cb022dd into main Sep 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant