From e1dafd31d90d408f2eb3452136e77d6e22f165c7 Mon Sep 17 00:00:00 2001 From: xwings Date: Wed, 23 Sep 2026 22:51:16 +0800 Subject: [PATCH] updated ARCHITECTURE files --- ARCHITECTURE.md | 4 +++- ARCHITECTURE/AGENT_RULES.md | 2 +- ARCHITECTURE/indexes/firmware.md | 2 +- ARCHITECTURE/indexes/operating-systems.md | 2 +- ARCHITECTURE/indexes/runtime.md | 2 +- ARCHITECTURE/modules/arch.md | 18 ++++++++++++------ ARCHITECTURE/modules/baremetal.md | 9 +++++---- ARCHITECTURE/modules/cli-build.md | 12 +++++++----- ARCHITECTURE/modules/core.md | 8 ++++++-- ARCHITECTURE/modules/debugger.md | 2 +- ARCHITECTURE/modules/dos.md | 6 +++--- ARCHITECTURE/modules/extensions.md | 4 ++-- ARCHITECTURE/modules/hardware.md | 2 +- ARCHITECTURE/modules/kernel-proxy.md | 17 +++++++++++------ ARCHITECTURE/modules/loaders.md | 6 +++--- ARCHITECTURE/modules/os-base.md | 2 +- ARCHITECTURE/modules/posix.md | 22 ++++++++++++++++++---- ARCHITECTURE/modules/uefi.md | 6 +++--- ARCHITECTURE/modules/windows.md | 2 +- 19 files changed, 81 insertions(+), 47 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index fd0167e60..38619683c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,9 +1,11 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Qiling Architecture +Generated with [eatmycode](https://github.com/xwings/eatmycode). + ## Read First Before planning code changes or reviewing code, read diff --git a/ARCHITECTURE/AGENT_RULES.md b/ARCHITECTURE/AGENT_RULES.md index ab0bd43f4..ef6fba734 100644 --- a/ARCHITECTURE/AGENT_RULES.md +++ b/ARCHITECTURE/AGENT_RULES.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Agent Rules diff --git a/ARCHITECTURE/indexes/firmware.md b/ARCHITECTURE/indexes/firmware.md index e38288b96..fcf2bbd59 100644 --- a/ARCHITECTURE/indexes/firmware.md +++ b/ARCHITECTURE/indexes/firmware.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Firmware Routes diff --git a/ARCHITECTURE/indexes/operating-systems.md b/ARCHITECTURE/indexes/operating-systems.md index baeadd895..c12a2742f 100644 --- a/ARCHITECTURE/indexes/operating-systems.md +++ b/ARCHITECTURE/indexes/operating-systems.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Operating System Routes diff --git a/ARCHITECTURE/indexes/runtime.md b/ARCHITECTURE/indexes/runtime.md index 83cf8caf5..441debe20 100644 --- a/ARCHITECTURE/indexes/runtime.md +++ b/ARCHITECTURE/indexes/runtime.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Runtime Routes diff --git a/ARCHITECTURE/modules/arch.md b/ARCHITECTURE/modules/arch.md index 608ad1423..9ec2f50bb 100644 --- a/ARCHITECTURE/modules/arch.md +++ b/ARCHITECTURE/modules/arch.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # CPU Architecture and Calling Conventions @@ -21,7 +21,7 @@ or cross-platform compatibility certification. | --- | --- | | [arch/arch.py](../../qiling/arch/arch.py): `QlArch` | CPU, pointer width, stack, context interface | | [arch/register.py](../../qiling/arch/register.py): `QlRegisterManager` | Name-to-Unicorn registers and PC/SP aliases | -| [arch/](../../qiling/arch/), [arch/models.py](../../qiling/arch/models.py) | Concrete x86/ARM/MIPS/RISC-V/PPC/Cortex-M adapters and CPU enums | +| [arch/](../../qiling/arch/), [arch/models.py](../../qiling/arch/models.py), `*_const.py` | Concrete x86/ARM/MIPS/RISC-V/PPC/Cortex-M adapters, CPU enums, register maps and `EXCP` exception codes | | [cc/__init__.py](../../qiling/cc/__init__.py): `QlCC`, `QlCommonBaseCC` | Register/stack slots, result, return address, frame unwind | | [arch/cortex_m.py](../../qiling/arch/cortex_m.py): `QlArchCORTEX_M` | Exception/vector state and task-aware Unicorn | | [test_cpu_models.py](../../tests/test_cpu_models.py), [test_riscv.py](../../tests/test_riscv.py), [test_shellcode.py](../../tests/test_shellcode.py) | Model selection and instruction/ABI samples | @@ -43,8 +43,13 @@ versions come from the root manifest, not individual adapters. multiword arguments, shadow space, register exhaustion, stack return addresses and unwind rules differ (`cc/__init__.py`, `cc/intel.py`). - Endianness, CPU mode and Thumb PC handling affect execution and instruction - decoding. Preserve `effective_pc` and Cortex-M exception return behavior - when changing ARM mode; inspect the core Thumb workaround too. + decoding. Preserve the ARM adapter's `effective_pc` (consumers fall back + to `arch_pc` via `getattr`) and Cortex-M exception return behavior when + changing ARM mode; inspect the core Thumb workaround too. +- Interrupt numbers delivered to `hook_intno` are QEMU exception codes. + **Observed:** ARM/ARM64 and MIPS Linux traps use the `EXCP` enums in + `cortex_m_const.py` and `mips_const.py`; x86/RISC-V/PPC still pass + literals. Prefer an existing enum and extend it for new codes. - Concrete adapters instantiate engine/assembler/disassembler objects; supported enums do not imply support for every OS/CPU combination. Cortex-M uses `MultiTaskUnicorn` and hardware interrupt state. @@ -73,8 +78,9 @@ never substitute host pointer size for a guest ABI. From `tests/`: `python -m unittest test_cpu_models test_riscv test_shellcode` with root dependencies and relevant rootfs samples. All selected modules -passed during rebuild; CPU-model suite has 7 tests. Their scope is selected -models/instructions and guest execution, not instruction-set conformance. +passed during the latest refresh (11 CPU-model/RISC-V and 8 shellcode +cases). Their scope is selected models/instructions and guest execution, +not instruction-set conformance. For PPC changes use `test_elf.ELFTest.test_elf_linux_powerpc`; for ABI changes run the consuming OS API/syscall regression as well. diff --git a/ARCHITECTURE/modules/baremetal.md b/ARCHITECTURE/modules/baremetal.md index 65c593c7a..b97b2f599 100644 --- a/ARCHITECTURE/modules/baremetal.md +++ b/ARCHITECTURE/modules/baremetal.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Bare-metal Execution @@ -71,8 +71,8 @@ semantics into unrelated POSIX/Windows run loops. ## Verification From `tests/`: `python -m unittest test_mcu.MCUTest.test_mcu_snapshot_stm32f411 test_mcu.MCUTest.test_mcu_usart_input_stm32f411 test_blob.BlobTest.test_uboot_arm` -passed 3 cases during rebuild. Root dependencies and matching STM32/U-Boot -fixtures are required. Run `python test_mcu.py` for broader device/timing +passed 3 cases during the latest refresh. Root dependencies and matching +STM32/U-Boot fixtures are required. Run `python test_mcu.py` for broader device/timing changes and `python test_blob.py` for raw behavior; the full suites are not certified by the selected cases. @@ -80,7 +80,8 @@ certified by the selected cases. `test_blob.BlobTest.test_blob_raw` fails because rootfs lacks `blob/example_raw.bin`; [fixture source](../../examples/src/blob/Makefile) -exists but was not built or copied into the submodule during this rebuild. +exists but is not built or copied into the submodule; the case still +errors with a missing-file error. Fast-mode faults currently stop cleanly instead of delivering hardware HardFault. Timeout units differ by execution path; full scheduler timing and MCU exception fidelity remain unverified. diff --git a/ARCHITECTURE/modules/cli-build.md b/ARCHITECTURE/modules/cli-build.md index 89b5bd752..65325dbbb 100644 --- a/ARCHITECTURE/modules/cli-build.md +++ b/ARCHITECTURE/modules/cli-build.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # CLI, Packaging and Development Tooling @@ -95,10 +95,12 @@ CLI/version/profile behavior and TUI import outside the checkout. For checkout behavior from `tests/`: `python test_qltool.py`; rootfs samples are required for argument, JSON, filter and coverage cases. -Rebuild evidence: sdist plus wheel-from-sdist build passed, artifact metadata -checks and `poetry check --lock` passed, 8 selected CLI tests passed, and 3 installed-wheel cases -passed in a second fresh environment. See [root verification](../../ARCHITECTURE.md#verification) -for common setup and Linux aggregate. No configured lint/type-check command exists. +Latest refresh evidence: sdist plus wheel-from-sdist build, `twine check +--strict` and `poetry check --lock` passed; 5 checkout CLI cases and 3 +installed-wheel cases passed in a second fresh environment; the wheel +contains `qltui.py`, the 7 profiles and 43 GDB XML files. See +[root verification](../../ARCHITECTURE.md#verification) for common setup +and Linux aggregate. No configured lint/type-check command exists. ## Known Gaps diff --git a/ARCHITECTURE/modules/core.md b/ARCHITECTURE/modules/core.md index a8acf879b..6ebaae690 100644 --- a/ARCHITECTURE/modules/core.md +++ b/ARCHITECTURE/modules/core.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Core Runtime @@ -45,6 +45,9 @@ by `hookcallback` and re-raised by `emu_start`, not silently consumed. - Factories derive modules/class names from enums with explicit format/arch mappings. A new enum alone is insufficient (`utils.py`). Hook callback signatures and `QL_HOOK_BLOCK` behavior must survive dispatcher changes. + Interrupt, memory-fault and invalid-instruction events that no hook + handles raise `QlErrorCoreHook` (`core_hooks.py`); OS personalities + choose which interrupt numbers they hook. - Profiles use defaults plus OS/user overrides; MCU uses YAML-derived maps (`profile_setup`). Parsing belongs here; the consumer owns each key. - `save` includes only selected components. `restore` applies only included @@ -80,7 +83,8 @@ not a claim that the tree is an acyclic import graph. From `tests/`, after root setup: `python -m unittest test_shellcode`; `python -m unittest test_elf.ELFTest.test_memory_search` checks memory-facing -facade behavior. Both passed during rebuild on Linux/Python 3.13/Unicorn 2.1.3. +facade behavior. Both passed during the latest refresh on Linux/Python +3.13/Unicorn 2.1.3. For snapshot changes use matching cases in `test_elf.py` and `test_mcu.MCUTest.test_mcu_snapshot_stm32f411` (the MCU case also passed). Fixtures are relative to `tests/`. Passing these does not certify every diff --git a/ARCHITECTURE/modules/debugger.md b/ARCHITECTURE/modules/debugger.md index 5cc7cb443..34547df56 100644 --- a/ARCHITECTURE/modules/debugger.md +++ b/ARCHITECTURE/modules/debugger.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Debuggers diff --git a/ARCHITECTURE/modules/dos.md b/ARCHITECTURE/modules/dos.md index 4f4fdaa6b..7d692f8ab 100644 --- a/ARCHITECTURE/modules/dos.md +++ b/ARCHITECTURE/modules/dos.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # DOS and BIOS Interrupts @@ -69,8 +69,8 @@ not apply to DOS interrupt leaves. ## Verification From `tests/`: `python -m unittest test_dos` passed one program test during -rebuild with the existing DOS rootfs fixture. Use `python test_dos_exe.py` -for EXE-loader changes; it was not part of the verified subset. Terminal +the latest refresh with the existing DOS rootfs fixture. Use +`python test_dos_exe.py` for EXE-loader changes; it was not part of the verified subset. Terminal cases need usable curses/TTY support. A successful sample does not certify all BIOS services or error leaves. diff --git a/ARCHITECTURE/modules/extensions.md b/ARCHITECTURE/modules/extensions.md index 7f31a30e1..b10ca529b 100644 --- a/ARCHITECTURE/modules/extensions.md +++ b/ARCHITECTURE/modules/extensions.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Instrumentation and Integrations @@ -77,7 +77,7 @@ From `tests/`: `python -m unittest test_history` passed 4 tests; `python test_uefi.py` passed 2 direct integration tests including sanitizer behavior. For coverage output run `python -m unittest test_qltool.Qltool_Test.test_qltool_coverage` with UEFI -fixtures; that coverage case also passed during rebuild. For r2 changes use `python test_r2.py` after root setup with `RE`; +fixtures; that coverage case also passed during the latest refresh. For r2 changes use `python test_r2.py` after root setup with `RE`; this optional check was not run. AFL/IDA require their external runtimes and matching harnesses; a base-package import does not validate them. diff --git a/ARCHITECTURE/modules/hardware.md b/ARCHITECTURE/modules/hardware.md index 5bc0feb7e..3bbad01e6 100644 --- a/ARCHITECTURE/modules/hardware.md +++ b/ARCHITECTURE/modules/hardware.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Hardware Models and MMIO diff --git a/ARCHITECTURE/modules/kernel-proxy.md b/ARCHITECTURE/modules/kernel-proxy.md index 3852fcda8..fe8846920 100644 --- a/ARCHITECTURE/modules/kernel-proxy.md +++ b/ARCHITECTURE/modules/kernel-proxy.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Linux Kernel Proxy @@ -12,7 +12,8 @@ Read when: changing qiling/os/posix/kernel_proxy/, forwarded syscalls, proxy des Owns opt-in forwarding of selected Linux syscalls through a helper process, argument/buffer transport and proxy-owned descriptors. **In progress:** -Phase 0 code exists and 21 tests passed, but actual guest dispatch and +Phase 0 code exists; 20 of 21 tests pass and the pipe2 write-back case +errors on descriptor cleanup (Known Gaps). Actual guest dispatch and cross-ABI buffer semantics have unresolved gaps. Later phases in [TODO.md](../../TODO.md) remain designs, not implemented capability. @@ -74,8 +75,10 @@ No additional runtime package is required beyond the existing manifest. ## Verification -From `tests/`: `python -m unittest test_kernel_proxy` passed 21 tests on -Linux x86_64/Python 3.13 with current hello fixtures. Helpers invoke real +From `tests/`: `python -m unittest test_kernel_proxy` ran 21 tests on +Linux x86_64 (WSL2)/Python 3.13 with current hello fixtures during the +latest refresh: 20 passed and `test_ptr_out_writes_back_to_guest_memory` +errored with `EBADF` after its assertions passed. Helpers invoke real Linux syscalls. These tests exercise IPC and direct hook invocation; they do not establish complete guest-trap integration or cross-architecture struct conversion. Run an actual emulated syscall case for dispatch changes. @@ -85,8 +88,10 @@ struct conversion. Run an actual emulated syscall case for dispatch changes. `KernelProxy._make_forwarder` returns `(ql, *args)`, while POSIX dispatch counts only ordinary positional parameters; tests calling the hook directly can miss missing guest arguments. `test_ptr_out_writes_back_to_guest_memory` -closes helper-owned fd numbers with parent `os.close`; its passing result -can depend on coincident parent fd allocation and does not prove cleanup. +closes helper-owned fd numbers with parent `os.close`; depending on parent +fd allocation this fails with `EBADF` or closes an unrelated parent +descriptor. Its marshalling assertions pass; fix cleanup to go through the +proxy fd wrapper or the helper before treating the suite as green. Buffer quotas, ABI translation and concurrent IPC access need evidence; none should be implied by Phase 0 passing tests. Preserve scope and record required fixes before claiming later TODO milestones. diff --git a/ARCHITECTURE/modules/loaders.md b/ARCHITECTURE/modules/loaders.md index 55ff3bcb9..ae3bdd419 100644 --- a/ARCHITECTURE/modules/loaders.md +++ b/ARCHITECTURE/modules/loaders.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Executable Loaders @@ -75,8 +75,8 @@ OS imports in loaders are intentional construction dependencies. ## Verification From `tests/`: `python -m unittest test_elf.ELFTest.test_elf_linux_x8664 test_blob.BlobTest.test_uboot_arm` -and `python test_uefi.py`. These passed during rebuild with local fixtures. -Use direct UEFI script execution: its test bodies are guarded by `__main__`. +and `python test_uefi.py`. These passed during the latest refresh with +local fixtures. Use direct UEFI script execution: its test bodies are guarded by `__main__`. For changed formats, use the corresponding test files in Code Map; Windows and macOS need their system-library fixtures. Passing ELF does not verify PE. diff --git a/ARCHITECTURE/modules/os-base.md b/ARCHITECTURE/modules/os-base.md index 925c23b5c..26c6c065f 100644 --- a/ARCHITECTURE/modules/os-base.md +++ b/ARCHITECTURE/modules/os-base.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Shared Operating System Services diff --git a/ARCHITECTURE/modules/posix.md b/ARCHITECTURE/modules/posix.md index 07a5b80df..5b6f44734 100644 --- a/ARCHITECTURE/modules/posix.md +++ b/ARCHITECTURE/modules/posix.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # POSIX Personalities and Syscalls @@ -53,6 +53,15 @@ Fixture Makefiles specify cross-compilers locally; no common C standard is decla - Linux `/proc/self` mappings are installed only when not overridden. Profile identity and network keys affect observable guest behavior. macOS kernel and QNX message interfaces retain their own state/layouts. +- Linux registers `hook_intno` traps per architecture: the syscall + interrupt (via `EXCP` enums on ARM/ARM64/MIPS, literals elsewhere) plus + ARM/ARM64 `UDEF` and MIPS `RI` CPU exceptions. + `hook_cpu_exception` logs the SIGILL equivalent and calls `ql.stop()`, + so falling into non-code bytes ends emulation cleanly instead of raising + `QlErrorCoreHook`; no guest signal handler runs (`linux/linux.py`). +- Syscall result structs follow `ql.arch.endian`: `statx` selects the + `*EB` big-endian ctypes variants for EB guests (`syscall/stat.py`). + Check endian handling in any handler that writes ctypes layouts. ## Dependencies and Boundaries @@ -68,14 +77,17 @@ own dispatch. Filesystem/socket/fork operations expose host resources. | --- | --- | --- | | New/fixed syscall | Mapping table, ABI, OS override/common handler | Update this contract; matching `test_posix`/ELF test and affected OS | | Thread/futex/signal | Thread contexts, yielding, futex waiters | `test_elf_multithread.py`; mark unsupported semantics explicitly | +| CPU exception/trap | `hook_intno` registrations, `hook_cpu_exception`, `EXCP` enums | Shellcode illegal-instruction cases; read arch for new exception codes | | FD/path/socket | Host object lifetime, errno, guest structure layouts | OS-base checks and syscall regression; proxy checks if shared operations change | | macOS/QNX/kernel API | Local maps/structs, loader imports | Platform-specific test and loader owner | ## Verification From `tests/`: `python -m unittest test_elf.ELFTest.test_elf_linux_x8664 test_riscv test_qnx` -passed during rebuild. It checks sample Linux/RISC-V/QNX flows, not every -handler. For POSIX changes run `python test_posix.py` (also imports ELF, +and `python -m unittest test_shellcode.TestShellcode.test_linux_arm64_illegal_instruction test_shellcode.TestShellcode.test_linux_mips32_illegal_instruction test_elf.ELFTest.test_linux_statx_bigendian` +passed during the latest refresh. They check sample Linux/RISC-V/QNX flows, +SIGILL-style termination and big-endian `statx`, not every handler. +For POSIX changes run `python test_posix.py` (also imports ELF, RISC-V and CLI suites), and the relevant thread/kernel/network/platform script. Follow [shared test-resource rules](cli-build.md#verification) when running these suites. Root Linux aggregate is the broader CI gate; its fixtures, extracted @@ -84,7 +96,9 @@ kernel sample and network/host requirements apply. Full aggregate was not run. ## Known Gaps macOS CI is commented out. Some thread classes are unset for RISC-V/PPC; -signal syscall state is not proof of full delivery. Existing kernel-module +signal syscall state is not proof of full delivery. CPU-exception hooks +exist only for ARM/ARM64/MIPS; x86, RISC-V and PPC undefined instructions +still surface as `QlErrorCoreHook`. Existing kernel-module archives need fixture preparation. `TODO.md` hybrid phases are plans; proxy Phase 0 implementation is documented separately. Current representative checks do not establish kernel fidelity or host isolation. diff --git a/ARCHITECTURE/modules/uefi.md b/ARCHITECTURE/modules/uefi.md index 8abee70a2..ea8e3eee2 100644 --- a/ARCHITECTURE/modules/uefi.md +++ b/ARCHITECTURE/modules/uefi.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # UEFI Services @@ -68,8 +68,8 @@ UEFI shares PE parsing infrastructure, not Windows API semantics. ## Verification From `tests/`: **`python test_uefi.py`** with root dependencies and -`examples/rootfs/x8664_efi` fixtures. Both tests passed during rebuild; -they exercise interception and sanitized heap behavior. Do not substitute +`examples/rootfs/x8664_efi` fixtures. Both tests passed during the latest +refresh; they exercise interception and sanitized heap behavior. Do not substitute `python -m unittest test_uefi`: actual bodies are guarded by `__main__`. For package/coverage output changes also use the CLI coverage test through [extensions](extensions.md). diff --git a/ARCHITECTURE/modules/windows.md b/ARCHITECTURE/modules/windows.md index 33ff4918f..995d6a1ac 100644 --- a/ARCHITECTURE/modules/windows.md +++ b/ARCHITECTURE/modules/windows.md @@ -1,5 +1,5 @@ --- -eatmycode_version: "2.0.0" +eatmycode_version: "2.1.0" --- # Windows API Emulation