diff --git a/architecture.md b/architecture.md index a17f590a..a3811e52 100644 --- a/architecture.md +++ b/architecture.md @@ -293,7 +293,8 @@ MCP Client → Calls → Open in Report Viewer → ReportViewerGUI.show_report(M `event_dict` maps every keyword to a callable, on `execute_action()` reading an object-shaped script's actions from the key `webdriver_wrapper`, `auto_control` or `load_density`, and on their own keywords starting `WR_`, `AC_` or `LD_` (`framework_profiles.py`). `test_metadata_probe.py` asks the installed - packages and reads their executors' source for the key. + packages and exercises their public `execute_files()` APIs in isolated child processes to verify + the profile's document key, bare action lists, and rejection of unrelated keys. - **The automation packages' reports**: the report viewer reads what `je_api_testka`, `je_auto_control`, `je_web_runner` and `je_load_density` write (`execution_report/record_reports.py`) and relies on the pair of files `_success.json` / `_failure.json` (or `.xml`, under an `` root), on their diff --git a/architecture_explore.md b/architecture_explore.md index 9c1ffcb9..8f703bf2 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -416,6 +416,8 @@ first_summary → first_code_review → judge_single_review ┐(評分前一 **動作腳本的語言服務** `language_service/`(`docs/adr/0010`):WebRunner、AutoControl、LoadDensity 的腳本形狀相同(JSON 的動作清單,`["關鍵字"]` 或 `["關鍵字", 引數]`;整份是清單,或放在框架自己的 key 底下),所以一個 adapter 服務三個框架。 +`test_metadata_probe.py` 在獨立子程序驗證這三個框架的文件契約:把執行器的指令表換成只記錄引數的測試函式,透過公開的 `execute_files()` 讀取 JSON,檢查 profile 指定的 key、直接使用清單,以及拒絕錯誤 key。物件同時帶有無關 key,驗證執行器確實選取正確清單;不依賴執行器原始碼使用 `.get()` 或共用解析器,也不在測試主程序匯入框架。 + | 模組 | 內容 | |---|---| | `service_adapter.py`(271) | Phase 0 的契約(`docs/adr/0004`):LSP 形狀的型別、`LanguageServiceAdapter`、只問 adapter 說自己會的事並把它的例外留在裡面的 `LanguageService`、`LanguageServiceRegistry` | diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 7dfe22ba..28d4557c 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -327,3 +327,12 @@ Validation: 3514 passed, 17 skipped in test/test_utils/; Ruff reports no finding - **Completed**: the platform-smoke verification portion of `progress.md` #125. Both Linux and macOS jobs pass on PR #142 at `bb87f28`; Linux supplies Xvfb, xauth and libpulse0. Windows tests remain the full-suite gate. - **Files**: `progress.md` and the update-log index. - **Open items**: complete Linux/macOS unit suites and release gates remain in #125. + +## U-20261012-01 · 2026-10-12 · Verify framework document keys through their public executors · #incident #tests #compatibility + +- **Failure**: every Windows CI leg failed because the LoadDensity executor no longer contains the literal `.get("load_density"` after moving dispatch to ActionCore. Its document key remains `load_density`; source spelling was not the contract. +- **Fix**: replace the source-text assertion with an isolated child-process contract test for each installed framework. Replace its command registry with a recording function and execute a temporary JSON file through public `execute_files()`. Check the profile's key with an unrelated key beside it, a bare action list, and rejection of a document with only the unrelated key. The parent process never imports an automation framework, and no real automation command runs. +- **Checks**: the complete local unit suite passes (5036 passed, 4 skipped) with the previously installed framework versions, including all 28 metadata-probe tests. The four LoadDensity cases also pass against its current source and ActionCore's shared parser. All 28 tests also pass in a fresh environment with LoadDensity 0.0.79, AutoControl 1.0.4 and WebRunner 0.0.95; `pip check` finds no broken requirements. Pinned Ruff 0.15.22 and `git diff --check` pass. +- **Files**: `test/test_utils/test_metadata_probe.py`, `architecture.md`, `architecture_explore.md`, and this update-log index. +- **Analysis follow-up**: split the existing combined built-in assertion into separate assertions after SonarCloud reported it through its GitHub check annotations. The organization plan prevents its API from returning data for a PR targeting `dev`; the GitHub annotations provide the finding without exposing credentials. +- **Open items**: none for this incident. diff --git a/docs/updates/README.md b/docs/updates/README.md index fe4dbdf8..7455ba8b 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261012-01 | 2026-10-12 | Verify framework document keys through their public executors | #incident #tests #compatibility | [2026-10](2026-10.md) | | U-20261008-20 | 2026-10-08 | Linux and macOS platform smoke checks pass | #done #ci #platforms | [2026-10](2026-10.md) | | U-20261008-19 | 2026-10-08 | Keep FileAutomation backends installed after its extras split | #done #dependencies | [2026-10](2026-10.md) | | U-20261008-18 | 2026-10-08 | Name independent deterministic results and normalize hosts with HTTPS | #done #tests #security | [2026-10](2026-10.md) | @@ -722,4 +723,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here | [2026-09-c.md](2026-09-c.md) | 2026-09 | 115 | | [2026-09-d.md](2026-09-d.md) | 2026-09 | 91 | | [2026-09-e.md](2026-09-e.md) | 2026-09 | 83 | -| [2026-10.md](2026-10.md) | 2026-10 | 22 | +| [2026-10.md](2026-10.md) | 2026-10 | 23 | diff --git a/pyproject.toml b/pyproject.toml index d3a118be..75a0cdbe 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta" [project] name = "pybreeze" -version = "1.0.35" +version = "1.0.36" authors = [ { name = "JE-Chen", email = "jechenmailman@gmail.com" }, ] diff --git a/test/test_utils/test_metadata_probe.py b/test/test_utils/test_metadata_probe.py index 1c99bb1d..9fcbe7af 100644 --- a/test/test_utils/test_metadata_probe.py +++ b/test/test_utils/test_metadata_probe.py @@ -9,6 +9,8 @@ import ast import importlib.util +import json +import subprocess import sys from pathlib import Path @@ -18,6 +20,35 @@ from pybreeze.utils.language_service.framework_profiles import PROFILES, FrameworkProfile from pybreeze.utils.language_service.keyword_metadata import ParameterKind from pybreeze.utils.language_service.metadata_probe import PROBE_SCRIPT, probe_command, read_metadata +from pybreeze.utils.subprocess_util import utf8_subprocess_env + + +_EXECUTOR_MARKER = "pybreeze-executor-contract:" +_EXECUTOR_SCRIPT = r''' +import contextlib +import importlib +import json +import sys + +calls = [] + + +def record(value): + calls.append(value) + return value + + +error = None +with contextlib.redirect_stdout(sys.stderr): + try: + module = importlib.import_module(sys.argv[1]) + module.executor.event_dict = {"pybreeze_contract_probe": record} + module.execute_files([sys.argv[2]]) + except Exception as exc: + error = type(exc).__name__ + print(repr(exc), file=sys.stderr) +print(sys.argv[3] + json.dumps({"calls": calls, "error": error})) +''' FAKE = FrameworkProfile( framework="fake_runner", label="FakeRunner", document_key="fake_runner", executor_module="fake_runner.executor", @@ -122,7 +153,8 @@ def test_a_method_is_described_without_its_self(installed): def test_pythons_built_ins_are_marked_and_what_is_not_callable_is_left_out(installed): metadata = read_metadata(FAKE) - assert metadata.keyword("print").builtin and metadata.keyword("len").builtin + assert metadata.keyword("print").builtin + assert metadata.keyword("len").builtin assert not metadata.keyword("FR_open").builtin assert metadata.keyword("FR_table") is None @@ -240,15 +272,31 @@ def test_each_framework_lists_its_keywords_under_its_prefix(profile): @pytest.mark.parametrize("profile", PROFILES, ids=lambda profile: profile.framework) -def test_each_frameworks_executor_reads_its_actions_from_the_key_the_profile_names(profile): - # Found without importing the package: je_auto_control changes the process as it is imported - package = importlib.util.find_spec(profile.framework) - if package is None: +@pytest.mark.parametrize("document_form", ["wrapped", "list", "wrong-key"]) +def test_each_frameworks_executor_reads_its_actions_from_the_key_the_profile_names( + profile, document_form, tmp_path): + # Import and execute in a child: AutoControl changes the process on import. + if importlib.util.find_spec(profile.framework) is None: pytest.skip(f"{profile.framework} is not installed here") - module_path = Path(package.submodule_search_locations[0]).joinpath( - *profile.executor_module.split(".")[1:]).with_suffix(".py") - - source = module_path.read_text(encoding="utf-8") - - assert f'.get("{profile.document_key}"' in source - assert "event_dict" in source and "executor = Executor()" in source + actions = [["pybreeze_contract_probe", ["expected"]]] + document = {"pybreeze_unrelated_actions": [["pybreeze_contract_probe", ["wrong"]]]} + if document_form == "wrapped": + document[profile.document_key] = actions + elif document_form == "list": + document = actions + action_file = tmp_path / "actions.json" + action_file.write_text(json.dumps(document), encoding="utf-8") + result = subprocess.run( + [sys.executable, "-c", _EXECUTOR_SCRIPT, profile.executor_module, + str(action_file), _EXECUTOR_MARKER], + cwd=Path(sys.executable).parent, env=utf8_subprocess_env(), + capture_output=True, text=True, encoding="utf-8", timeout=60, shell=False, + check=False) + assert result.returncode == 0, result.stderr + answer = json.loads(result.stdout.removeprefix(_EXECUTOR_MARKER)) + if document_form == "wrong-key": + assert answer["error"] is not None, result.stderr + assert answer["calls"] == [], result.stderr + else: + assert answer["error"] is None, result.stderr + assert answer["calls"] == ["expected"], result.stderr