Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,8 @@ MCP Client → Calls → Open in Report Viewer → ReportViewerGUI.show_report(M
`event_dict` maps every keyword to a callable, on `execute_action()` reading an object-shaped script's
actions from the key `webdriver_wrapper`, `auto_control` or `load_density`, and on their own keywords
starting `WR_`, `AC_` or `LD_` (`framework_profiles.py`). `test_metadata_probe.py` asks the installed
packages and reads their executors' source for the key.
packages and exercises their public `execute_files()` APIs in isolated child processes to verify
the profile's document key, bare action lists, and rejection of unrelated keys.
- **The automation packages' reports**: the report viewer reads what `je_api_testka`, `je_auto_control`,
`je_web_runner` and `je_load_density` write (`execution_report/record_reports.py`) and relies on the pair of
files `<name>_success.json` / `<name>_failure.json` (or `.xml`, under an `<xml_data>` root), on their
Expand Down
2 changes: 2 additions & 0 deletions architecture_explore.md
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,8 @@ first_summary → first_code_review → judge_single_review ┐(評分前一

**動作腳本的語言服務** `language_service/`(`docs/adr/0010`):WebRunner、AutoControl、LoadDensity 的腳本形狀相同(JSON 的動作清單,`["關鍵字"]` 或 `["關鍵字", 引數]`;整份是清單,或放在框架自己的 key 底下),所以一個 adapter 服務三個框架。

`test_metadata_probe.py` 在獨立子程序驗證這三個框架的文件契約:把執行器的指令表換成只記錄引數的測試函式,透過公開的 `execute_files()` 讀取 JSON,檢查 profile 指定的 key、直接使用清單,以及拒絕錯誤 key。物件同時帶有無關 key,驗證執行器確實選取正確清單;不依賴執行器原始碼使用 `.get()` 或共用解析器,也不在測試主程序匯入框架。

| 模組 | 內容 |
|---|---|
| `service_adapter.py`(271) | Phase 0 的契約(`docs/adr/0004`):LSP 形狀的型別、`LanguageServiceAdapter`、只問 adapter 說自己會的事並把它的例外留在裡面的 `LanguageService`、`LanguageServiceRegistry` |
Expand Down
9 changes: 9 additions & 0 deletions docs/updates/2026-10.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,3 +327,12 @@ Validation: 3514 passed, 17 skipped in test/test_utils/; Ruff reports no finding
- **Completed**: the platform-smoke verification portion of `progress.md` #125. Both Linux and macOS jobs pass on PR #142 at `bb87f28`; Linux supplies Xvfb, xauth and libpulse0. Windows tests remain the full-suite gate.
- **Files**: `progress.md` and the update-log index.
- **Open items**: complete Linux/macOS unit suites and release gates remain in #125.

## U-20261012-01 · 2026-10-12 · Verify framework document keys through their public executors · #incident #tests #compatibility

- **Failure**: every Windows CI leg failed because the LoadDensity executor no longer contains the literal `.get("load_density"` after moving dispatch to ActionCore. Its document key remains `load_density`; source spelling was not the contract.
- **Fix**: replace the source-text assertion with an isolated child-process contract test for each installed framework. Replace its command registry with a recording function and execute a temporary JSON file through public `execute_files()`. Check the profile's key with an unrelated key beside it, a bare action list, and rejection of a document with only the unrelated key. The parent process never imports an automation framework, and no real automation command runs.
- **Checks**: the complete local unit suite passes (5036 passed, 4 skipped) with the previously installed framework versions, including all 28 metadata-probe tests. The four LoadDensity cases also pass against its current source and ActionCore's shared parser. All 28 tests also pass in a fresh environment with LoadDensity 0.0.79, AutoControl 1.0.4 and WebRunner 0.0.95; `pip check` finds no broken requirements. Pinned Ruff 0.15.22 and `git diff --check` pass.
- **Files**: `test/test_utils/test_metadata_probe.py`, `architecture.md`, `architecture_explore.md`, and this update-log index.
- **Analysis follow-up**: split the existing combined built-in assertion into separate assertions after SonarCloud reported it through its GitHub check annotations. The organization plan prevents its API from returning data for a PR targeting `dev`; the GitHub annotations provide the finding without exposing credentials.
- **Open items**: none for this incident.
3 changes: 2 additions & 1 deletion docs/updates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here

| ID | Date | Title | Tags | Batch |
|---|---|---|---|---|
| U-20261012-01 | 2026-10-12 | Verify framework document keys through their public executors | #incident #tests #compatibility | [2026-10](2026-10.md) |
| U-20261008-20 | 2026-10-08 | Linux and macOS platform smoke checks pass | #done #ci #platforms | [2026-10](2026-10.md) |
| U-20261008-19 | 2026-10-08 | Keep FileAutomation backends installed after its extras split | #done #dependencies | [2026-10](2026-10.md) |
| U-20261008-18 | 2026-10-08 | Name independent deterministic results and normalize hosts with HTTPS | #done #tests #security | [2026-10](2026-10.md) |
Expand Down Expand Up @@ -722,4 +723,4 @@ In the same commit: delete the item from `progress.md`, add a `#done` entry here
| [2026-09-c.md](2026-09-c.md) | 2026-09 | 115 |
| [2026-09-d.md](2026-09-d.md) | 2026-09 | 91 |
| [2026-09-e.md](2026-09-e.md) | 2026-09 | 83 |
| [2026-10.md](2026-10.md) | 2026-10 | 22 |
| [2026-10.md](2026-10.md) | 2026-10 | 23 |
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "pybreeze"
version = "1.0.35"
version = "1.0.36"
authors = [
{ name = "JE-Chen", email = "jechenmailman@gmail.com" },
]
Expand Down
72 changes: 60 additions & 12 deletions test/test_utils/test_metadata_probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@

import ast
import importlib.util
import json
import subprocess
import sys
from pathlib import Path

Expand All @@ -18,6 +20,35 @@
from pybreeze.utils.language_service.framework_profiles import PROFILES, FrameworkProfile
from pybreeze.utils.language_service.keyword_metadata import ParameterKind
from pybreeze.utils.language_service.metadata_probe import PROBE_SCRIPT, probe_command, read_metadata
from pybreeze.utils.subprocess_util import utf8_subprocess_env


_EXECUTOR_MARKER = "pybreeze-executor-contract:"
_EXECUTOR_SCRIPT = r'''
import contextlib
import importlib
import json
import sys

calls = []


def record(value):
calls.append(value)
return value


error = None
with contextlib.redirect_stdout(sys.stderr):
try:
module = importlib.import_module(sys.argv[1])
module.executor.event_dict = {"pybreeze_contract_probe": record}
module.execute_files([sys.argv[2]])
except Exception as exc:
error = type(exc).__name__
print(repr(exc), file=sys.stderr)
print(sys.argv[3] + json.dumps({"calls": calls, "error": error}))
'''

FAKE = FrameworkProfile(
framework="fake_runner", label="FakeRunner", document_key="fake_runner", executor_module="fake_runner.executor",
Expand Down Expand Up @@ -122,7 +153,8 @@ def test_a_method_is_described_without_its_self(installed):
def test_pythons_built_ins_are_marked_and_what_is_not_callable_is_left_out(installed):
metadata = read_metadata(FAKE)

assert metadata.keyword("print").builtin and metadata.keyword("len").builtin
assert metadata.keyword("print").builtin
assert metadata.keyword("len").builtin
assert not metadata.keyword("FR_open").builtin
assert metadata.keyword("FR_table") is None

Expand Down Expand Up @@ -240,15 +272,31 @@ def test_each_framework_lists_its_keywords_under_its_prefix(profile):


@pytest.mark.parametrize("profile", PROFILES, ids=lambda profile: profile.framework)
def test_each_frameworks_executor_reads_its_actions_from_the_key_the_profile_names(profile):
# Found without importing the package: je_auto_control changes the process as it is imported
package = importlib.util.find_spec(profile.framework)
if package is None:
@pytest.mark.parametrize("document_form", ["wrapped", "list", "wrong-key"])
def test_each_frameworks_executor_reads_its_actions_from_the_key_the_profile_names(
profile, document_form, tmp_path):
# Import and execute in a child: AutoControl changes the process on import.
if importlib.util.find_spec(profile.framework) is None:
pytest.skip(f"{profile.framework} is not installed here")
module_path = Path(package.submodule_search_locations[0]).joinpath(
*profile.executor_module.split(".")[1:]).with_suffix(".py")

source = module_path.read_text(encoding="utf-8")

assert f'.get("{profile.document_key}"' in source
assert "event_dict" in source and "executor = Executor()" in source
actions = [["pybreeze_contract_probe", ["expected"]]]
document = {"pybreeze_unrelated_actions": [["pybreeze_contract_probe", ["wrong"]]]}
if document_form == "wrapped":
document[profile.document_key] = actions
elif document_form == "list":
document = actions
action_file = tmp_path / "actions.json"
action_file.write_text(json.dumps(document), encoding="utf-8")
result = subprocess.run(
[sys.executable, "-c", _EXECUTOR_SCRIPT, profile.executor_module,
str(action_file), _EXECUTOR_MARKER],
cwd=Path(sys.executable).parent, env=utf8_subprocess_env(),
capture_output=True, text=True, encoding="utf-8", timeout=60, shell=False,
check=False)
assert result.returncode == 0, result.stderr
answer = json.loads(result.stdout.removeprefix(_EXECUTOR_MARKER))
if document_form == "wrong-key":
assert answer["error"] is not None, result.stderr
assert answer["calls"] == [], result.stderr
else:
assert answer["error"] is None, result.stderr
assert answer["calls"] == ["expected"], result.stderr
Loading