From dc64903d3d8e44013ccfeab7674640a436d5b490 Mon Sep 17 00:00:00 2001 From: xwings Date: Sun, 13 Sep 2026 20:06:03 +0800 Subject: [PATCH] Rebuilt ARCHITECTURE.md and ARCHITECTURE/ with eatmycode 2.0.0 --- ARCHITECTURE.md | 604 +++------------------- ARCHITECTURE/AGENT_RULES.md | 187 +++++++ ARCHITECTURE/arch.md | 149 ------ ARCHITECTURE/cli.md | 113 ---- ARCHITECTURE/core.md | 176 ------- ARCHITECTURE/debugger.md | 146 ------ ARCHITECTURE/extensions.md | 138 ----- ARCHITECTURE/hw.md | 145 ------ ARCHITECTURE/indexes/firmware.md | 22 + ARCHITECTURE/indexes/operating-systems.md | 24 + ARCHITECTURE/indexes/runtime.md | 23 + ARCHITECTURE/kernel-proxy.md | 152 ------ ARCHITECTURE/loader.md | 161 ------ ARCHITECTURE/modules/arch.md | 86 +++ ARCHITECTURE/modules/baremetal.md | 86 +++ ARCHITECTURE/modules/cli-build.md | 110 ++++ ARCHITECTURE/modules/core.md | 95 ++++ ARCHITECTURE/modules/debugger.md | 84 +++ ARCHITECTURE/modules/dos.md | 82 +++ ARCHITECTURE/modules/extensions.md | 89 ++++ ARCHITECTURE/modules/hardware.md | 86 +++ ARCHITECTURE/modules/kernel-proxy.md | 92 ++++ ARCHITECTURE/modules/loaders.md | 89 ++++ ARCHITECTURE/modules/os-base.md | 94 ++++ ARCHITECTURE/modules/posix.md | 90 ++++ ARCHITECTURE/modules/uefi.md | 82 +++ ARCHITECTURE/modules/windows.md | 87 ++++ ARCHITECTURE/os-baremetal.md | 124 ----- ARCHITECTURE/os-base.md | 161 ------ ARCHITECTURE/os-posix.md | 168 ------ ARCHITECTURE/os-windows.md | 153 ------ examples/rootfs | 2 +- 32 files changed, 1582 insertions(+), 2318 deletions(-) create mode 100644 ARCHITECTURE/AGENT_RULES.md delete mode 100644 ARCHITECTURE/arch.md delete mode 100644 ARCHITECTURE/cli.md delete mode 100644 ARCHITECTURE/core.md delete mode 100644 ARCHITECTURE/debugger.md delete mode 100644 ARCHITECTURE/extensions.md delete mode 100644 ARCHITECTURE/hw.md create mode 100644 ARCHITECTURE/indexes/firmware.md create mode 100644 ARCHITECTURE/indexes/operating-systems.md create mode 100644 ARCHITECTURE/indexes/runtime.md delete mode 100644 ARCHITECTURE/kernel-proxy.md delete mode 100644 ARCHITECTURE/loader.md create mode 100644 ARCHITECTURE/modules/arch.md create mode 100644 ARCHITECTURE/modules/baremetal.md create mode 100644 ARCHITECTURE/modules/cli-build.md create mode 100644 ARCHITECTURE/modules/core.md create mode 100644 ARCHITECTURE/modules/debugger.md create mode 100644 ARCHITECTURE/modules/dos.md create mode 100644 ARCHITECTURE/modules/extensions.md create mode 100644 ARCHITECTURE/modules/hardware.md create mode 100644 ARCHITECTURE/modules/kernel-proxy.md create mode 100644 ARCHITECTURE/modules/loaders.md create mode 100644 ARCHITECTURE/modules/os-base.md create mode 100644 ARCHITECTURE/modules/posix.md create mode 100644 ARCHITECTURE/modules/uefi.md create mode 100644 ARCHITECTURE/modules/windows.md delete mode 100644 ARCHITECTURE/os-baremetal.md delete mode 100644 ARCHITECTURE/os-base.md delete mode 100644 ARCHITECTURE/os-posix.md delete mode 100644 ARCHITECTURE/os-windows.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b62a5e235..fd0167e60 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,541 +1,83 @@ --- -eatmycode_version: "1.2.0" +eatmycode_version: "2.0.0" --- -# Qiling Framework — Architecture +# Qiling Architecture -This is the control center for agent-readable architecture docs. -Cross-cutting facts live here; each subsystem is documented once in -`ARCHITECTURE/.md` (see the [Index](#index)). `CLAUDE.md` and -`AGENT.md` are symlinks to this file. +## Read First -## Mission and Constraints +Before planning code changes or reviewing code, read +[Agent Rules](ARCHITECTURE/AGENT_RULES.md). Follow the Task Index to the +owning module and read pages whose **Read when** trigger matches the task. +Load partner modules only for affected boundaries; never load the entire +ARCHITECTURE directory. Reuse unchanged pages already read in this session. +Check claims against source, configuration, and tests; they remain +authoritative. If a route or fact is missing or stale, inspect source and +repair the affected docs. For broad changes, work through owners in batches +and retain cross-owner constraints and verification evidence. -Qiling is a binary emulation framework: it emulates and sandboxes code in -an isolated environment across platforms and architectures. Built on -Unicorn Engine, it adds what raw CPU emulation lacks: operating-system -context (syscalls, APIs, filesystems, registries), executable-format -loading, and dynamic linking (`README.md:13-27`). +## Project Snapshot -Supported combinations are defined in code: - -- Architectures — `QL_ARCH` (`qiling/const.py:15`): 8086, x86, x86-64, - ARM, ARM64, Cortex-M, MIPS, RISC-V 32/64, PowerPC. -- Operating systems — `QL_OS` (`qiling/const.py:28`): Linux, FreeBSD, - macOS, Windows, UEFI, DOS, QNX, MCU (bare-metal), BLOB. -- Formats: ELF, PE, Mach-O, COM/EXE/MBR, Intel HEX / raw firmware; kernel - modules for Windows `.sys`, Linux `.ko`, macOS `.kext`. - -Observable behavior: `Qiling(argv, rootfs, …).run()` executes the target -with hooks at instruction, basic-block, memory, interrupt, syscall, and -API level; VM state save/restore; hot patching; GDB-server and built-in -Qdb debugging (with record/replay); AFL++ fuzzing; and an opt-in kernel -proxy that forwards chosen Linux syscalls to a real kernel. - -Constraints and non-goals: - -- Emulation is single-Unicorn, cooperative-thread; there is no preemptive - threading, no signal delivery, and networking uses host sockets - (`TODO.md:9-21`). Real threading and signals are proposals only. -- Windows and macOS emulation need host-collected, non-redistributable - system libraries; those suites are host-gated. -- Coverage of syscalls, Win32 APIs, and peripheral registers is - demand-driven; unrequested surface is out of scope (see deviations). -- `unicorn` is hard-pinned; upgrading it is a project-wide event. - -## Languages and Toolchain - -| Area | Language / tool | Evidence | -| ---- | --------------- | -------- | -| `qiling/`, `tests/`, `examples/*.py`, `qltool`, `qltui.py` | Python, declared `^3.10` | `pyproject.toml:37` | -| CI matrix | Python 3.11 and 3.13 on `windows-latest` and `ubuntu-latest` (four jobs); the ubuntu/3.13 job carries a `container: Docker` marker | `.github/workflows/build-ci.yml:13-18` | -| Packaging | Poetry 2 (`poetry-core>=2.0,<3.0`), version `1.4.12.dev0`, GPL-2.0-or-later; `qltool` console script | `pyproject.toml:4`, `:7`, `:33-34`, `:61-63` | -| Wheel/sdist build check | `poetry check --lock`, `python -m build`, `twine check --strict` | `.github/workflows/pythonpublish.yml:20-44` | -| Container | `python:3.13-slim-trixie` multi-stage Poetry wheel build | `Dockerfile:1`, `:18-23` | -| Editor config | 4-space indent, LF, UTF-8, final newline for `*.py` | `.editorconfig:6-12` | -| Fixture sources | C/asm under `examples/src/`, `examples/shellcodes/`, `examples/fuzzing/*/fuzz.c` (built out-of-tree; binaries live in the rootfs submodule) | `examples/src/linux/hello.c` | -| Test scripts | Bash (`tests/test_onlinux.sh`, `tests/test_macho.sh`), batch (`tests/test_pe.bat`) | `tests/test_onlinux.sh:7-20` | - -Runtime dependencies (`pyproject.toml:36-50`): `unicorn == 2.1.3` -(hard-pinned), `capstone ^5`, `keystone-engine ^0.9.2`, `pefile`, -`pyelftools`, `python-registry`, `gevent >=24.10`, `multiprocess`, -`pyyaml ^6`, `windows-curses` (Windows only), and the TUI trio -`python-fx`/`questionary`/`termcolor`. Extras: `fuzz` → `unicornafl`, -`fuzzercorn`; `RE` → `r2libr` (`pyproject.toml:57-59`). - -No formatter, linter, or type checker is configured in the tree -(inspected: `pyproject.toml`, no `setup.cfg`/`tox.ini`/`.flake8`/ -`.pre-commit-config.yaml`). Declared support is the `^3.10` range above; -the versions this refresh was verified with (Python 3.13.5, Poetry 2.1.2, -unicorn 2.1.3) are a local observation, not a repository constraint. +| Fact | Value and evidence | +| --- | --- | +| Purpose | Binary emulation over Unicorn with images, OS APIs, hooks and firmware; entry is `Qiling(...).run()` ([core](qiling/core.py)). Complete OS/device fidelity is outside implemented scope. | +| Language / package | Python `^3.10`; package `1.4.12.dev0`, GPL-2.0-or-later; Poetry backend `poetry-core>=2.0,<3.0` ([manifest](pyproject.toml)). Fixture C/C++/assembly toolchains are local. | +| Engines | Unicorn **2.1.3**, Capstone `^5`, Keystone `^0.9.2`; other dependencies/platform extras in the manifest and [lock](poetry.lock). | +| Compatibility | Arch/OS enums in [const.py](qiling/const.py), composition in [utils.py](qiling/utils.py); enums do not promise every combination. [CI](.github/workflows/build-ci.yml) declares Python 3.11/3.13 on Ubuntu/Windows; macOS job is commented out. | ## System Design -Layers, bottom to top by import direction; a layer imports only layers -below it (plus the `Qiling` facade type from `qiling/core.py`, which every -module imports for annotations): - -| Layer | Owner doc | Responsibility and state | -| ----- | --------- | ------------------------ | -| Arch | [arch.md](ARCHITECTURE/arch.md) | The single Unicorn instance, registers, stack, disassembler/assembler, calling conventions | -| Core | [core.md](ARCHITECTURE/core.md) | `Qiling` facade, composition order, hook engine, component factories, profiles, logging, exceptions | -| OS base | [os-base.md](ARCHITECTURE/os-base.md) | Memory manager/heap, fcall marshalling, rootfs-confined paths, fs mapper, fd objects, green-thread base | -| OS personalities | [os-posix.md](ARCHITECTURE/os-posix.md), [os-windows.md](ARCHITECTURE/os-windows.md), [os-baremetal.md](ARCHITECTURE/os-baremetal.md) | Syscall/API/interrupt dispatch and run loops; own guest process state | -| Loader | [loader.md](ARCHITECTURE/loader.md) | Parses the untrusted image, maps it, builds initial state, records image tables; imports the OS personality's structs, hooks, and API tables (`qiling/loader/elf.py:24-27`, `pe.py:22-26`) | -| Hardware | [hw.md](ARCHITECTURE/hw.md) | MMIO peripherals and chip maps for MCU mode | -| Optional | [debugger.md](ARCHITECTURE/debugger.md), [extensions.md](ARCHITECTURE/extensions.md), [kernel-proxy.md](ARCHITECTURE/kernel-proxy.md), [cli.md](ARCHITECTURE/cli.md) | Consume only the public `Qiling`/`QlOs` API | - -Rules a change must respect: - -- **Name-based composition.** Core resolves arch/loader/OS/component/ - debugger classes by deriving module and class names from enum names in - `qiling/utils.py:297-417`; concrete classes are never imported by core. -- **Downward imports only.** Documented exceptions: `qiling/arch/cortex_m.py:22` - and `qiling/os/mcu/mcu.py:11` import `qiling/extensions/multitask.py`, - `qiling/arch/utils.py:94` lazily imports r2, `qiling/arch/x86_utils.py:10` - imports `QlMemoryManager` from OS base for a type annotation, and - `qiling/cli.py:22-23` imports coverage/report. Do not add more. Optional - modules mostly use the public API but also reach into concrete types - (`qiling/debugger/gdb/gdb.py:38`, `qiling/extensions/tracing/formats/registers.py:2`); - that is downward and allowed. -- **Dispatch models.** POSIX intercepts syscalls by number - (`qiling/os/posix/posix.py:170`); Windows/UEFI intercept API calls by - IAT address (`qiling/os/windows/windows.py:172`); DOS intercepts - interrupts by `(intno, AH)` (`qiling/os/dos/dos.py:83`); MCU steps - instruction-by-instruction with peripherals (`qiling/os/mcu/mcu.py:53`). -- **User override protocol.** `set_syscall`/`set_api` with - `QL_INTERCEPT.CALL|ENTER|EXIT` (`qiling/const.py:55`) is the only - supported way to replace or wrap emulated behavior; the kernel proxy is - built entirely on it. -- **Trust boundaries.** The guest is untrusted. Host exposure exists at: - rootfs path resolution (`qiling/os/path.py:239`), loader parsing of - header-derived sizes (`qiling/loader/`), host sockets and `os.fork` in - POSIX (`qiling/os/posix/syscall/sched.py:50-59`), and explicitly - forwarded syscalls in the kernel proxy. -- **Fidelity over abstraction.** Per-OS and per-chip code is deliberately - repetitive to mirror real platform behavior (see deviations). - -## Runtime and Data Flow - -1. Entry: `qltool` (checkout) or the installed console script call - `qiling.cli.run`, which ends in `Qiling(**ql_args)` (`qiling/cli.py:276`); - library users construct `Qiling` directly. -2. `Qiling.__init__` (`qiling/core.py:36`) guesses arch/OS from the file if - not given (`qiling/utils.py:278`), then composes in fixed order: arch → - struct/hook mixins → logger → profile → loader → memory manager → OS → - hardware manager (bare-metal only) → `loader.run()` → stop guard - (`qiling/core.py:154-197`). The target is fully mapped when the - constructor returns. -3. `Qiling.run()` (`qiling/core.py:561`) instantiates the debugger if set, - applies queued patches, writes the exit trap, and delegates to - `os.run()`; the debugger's `run()` follows. -4. The OS run loop drives `Qiling.emu_start` (`qiling/core.py:743`), the - thin wrapper over `uc.emu_start` that manages the thumb bit, `QL_STATE`, - and re-raises exceptions captured inside hooks. -5. Guest traps re-enter Python through `QlCoreHooks` dispatchers - (`qiling/core_hooks.py:167-276`) into the OS layer's syscall/API - handler, which reads arguments via the arch/ABI, executes, logs through - `QlOsUtils.print_function`, records stats, and writes the return value. -6. Exit: the guest reaches the OS exit point or exit trap, `emu_stop` is - called, `run()` returns; `ql.os.exit_code` carries the status. Unicorn - errors surface as `UcError` after `QlOs.emu_error` dumps context - (`qiling/os/os.py:249`). - -Configuration contract: INI profiles per OS in `qiling/profiles/.ql` -(sections such as `[OS32]/[OS64]`, `[CODE]`, `[KERNEL]`, `[MISC]`, -`[NETWORK]`; Windows adds `[PATH]`, `[USER]`, `[REGISTRY]`, …) merged with -a user path or dict (`qiling/utils.py:419-449`); MCU uses YAML merged into -`ql.env`. The `env` kwarg is the guest environment for POSIX/Windows and -the chip map for MCU. - -Persistence: none by default. `Qiling.save/restore` snapshot per-component -state (optional pickle file); the Windows registry writes hives back at -run end; `QlPeCache` caches parsed DLLs when `libcache=True`. - -Concurrency: one Unicorn per `Qiling`; POSIX/Windows threads are gevent -greenlets; MCU fast mode uses the cooperative `MultiTaskUnicorn`; `clone` -without `CLONE_VM` forks the host process; the kernel proxy is a child -process over a Unix socketpair. - -## Workspace Map - -| Path | Holds | -| ---- | ----- | -| `qiling/` | The framework package (owners in the Index) | -| `qiling/profiles/*.ql` | Default per-OS INI profiles | -| `qiling/os/posix/kernel_proxy/` | Kernel proxy (Linux-host optional feature) | -| `qiling/extensions/mcu/` | Chip `env` maps (owned by [hw.md](ARCHITECTURE/hw.md)) | -| `tests/` | Standalone `unittest` files run from `tests/`; CI drivers `test_onlinux.sh`, `test_pe.bat`, `test_macho.sh`; Qdb scripts in `qdb_scripts/`; test-only INI profiles in `profiles/`; scratch output in `log_test/` | -| `examples/` | Demo scripts, `fuzzing/`, `mcu/`, `shellcodes/`, `extensions/`, `scripts/` (DLL/dylib collectors), `src/` (fixture sources), and the `rootfs/` submodule (fixture binaries; do not edit in place, update the submodule) | -| `jexamples/` | Legacy examples, not covered by CI | -| `docs/` | One-line pointers to https://docs.qiling.io, images, and two Windows DLL inventory lists (`DLLX86.txt`, `DLLX8664.txt`); not a documentation source | -| `qltool`, `qiling/cli.py`, `qltui.py` | CLI launcher, implementation, TUI ([cli.md](ARCHITECTURE/cli.md)) | -| `pyproject.toml`, `poetry.lock` | The only manifest and lock file | -| `Dockerfile` | Container build of the wheel (build tooling; no runtime code) | -| `.github/workflows/` | `build-ci.yml` (tests), `pythonpublish.yml` (build checks + publish), `dockerimage.yml`, `giteesync.yml` | -| `TODO.md` | Hybrid-kernel design and the "Existing Issues" list; `ChangeLog` (stale at 1.4.6); `CREDITS.md`; `COPYING` | - -Generated or vendored: `examples/rootfs/` (git submodule, regenerate with -`git submodule update --init`); `qiling/os/uefi/Uefi*.py`, -`PiMultiPhase.py`, `ProcessorBind.py` are EDK2-derived type tables (edit -with care, no generator in tree); `qiling/debugger/gdb/xml/` mirrors GDB's -target descriptions. - -## Coding Style and Code Design - -No enforced formatter/linter/type checker exists; the rules below are -observed conventions with canonical implementations. - -- **Indentation**: 4 spaces (`.editorconfig:9-10`); every package file - indents with spaces except two that still contain tab-indented lines - (`qiling/os/posix/syscall/epoll.py:140-141`, - `qiling/debugger/qdb/branch_predictor/__init__.py:13-17`); do not add - more. -- **File header**: shebang plus the three-line framework banner - (`qiling/core.py:1-4`). -- **Typing**: public APIs are annotated; circular imports are avoided with - `from __future__ import annotations` and `TYPE_CHECKING` guards - (`qiling/log.py:6`, `:19-20`; used in 42/49 modules respectively). - Callback signatures are `Protocol`s (`qiling/core_hooks.py:55-134`). -- **Naming**: classes `Ql`; enums `QL_*` (`qiling/const.py`); - syscall handlers `ql_syscall_` (`qiling/os/posix/posix.py:19`); - Win32 hooks `hook_` with `@winsdkapi` - (`qiling/os/windows/dlls/kernel32/fibersapi.py:13-16`); UEFI `@dxeapi`; - peripherals `` with an inner `Type` struct - (`qiling/hw/char/stm32f4xx_usart.py:12`). -- **Errors**: raise `QlErrorBase` subclasses from `qiling/exception.py`; - unknown syscalls/APIs log a warning and raise only under `debug_stop` - (`qiling/os/posix/posix.py:255-258`, `qiling/os/windows/windows.py:196-199`). - `TODO.md:628-644` lists bare `except:` sites and `assert`-based - validation as known debt (18 bare `except:` remained at this refresh); - do not add new ones. -- **Logging**: always `ql.log`; `print` is reserved for Qdb/TUI/IDA and - guest console output. Stray diagnostic prints remain in - `qiling/extensions/multitask.py:328`, `:365`, `qiling/exception.py:90`, - `qiling/loader/macho.py:313`, `qiling/os/macos/kernel_api/kernel_api.py:1397`, - and `qiling/extensions/tracing/formats/tenet.py:64`; do not add more. - Syscall/API lines go through `QlOsUtils.print_function` - (`qiling/os/utils.py:106`). -- **Guest structs**: `ctypes` via `qiling/os/struct.py` factories - (`get_packed_struct`/`get_aligned_struct`) so endian/pointer width follow - the target (`qiling/os/posix/syscall/epoll.py:31-45`). -- **Component resolution**: never import concrete arch/OS/loader classes - in core; rely on the `select_*` factories (`qiling/utils.py:297-417`). -- **Tests**: one standalone `unittest` module per subsystem in `tests/`, - run from that directory with relative rootfs paths - (`tests/test_elf.py:146`); host-gated cases use `unittest.skipUnless` - (`tests/test_kernel_proxy.py:17`, `tests/test_pathutils.py:31`). -- **Docstrings**: Google-style `Args:`/`Returns:` on public methods - (`qiling/core.py:561-570`); comments explain workarounds with issue links - (`qiling/os/os.py:215-221`). - -## Verification and Review Map - -Setup (Linux, from a clone with the submodule): - -```sh -git submodule update --init # examples/rootfs fixtures -python3 -m pip install -e . # or: poetry install -cd examples/rootfs/x86_linux/kernel && unzip -P infected m0hamed_rootkit.ko.zip # only for test_elf_ko.py -``` - -All test commands run from `tests/`; a pass is unittest `OK` and exit 0. -Run suites one at a time: several bind fixed localhost ports -(`tests/test_elf.py:926`, `tests/test_tendaac15_httpd.py:98`, -`tests/test_debugger.py:112`); during this refresh a `test_posix.py` run -started alongside `test_elf.py` blocked in `epoll_wait` until killed. - -| Change area | Run | Owner doc | -| ----------- | --- | --------- | -| Core facade, hooks, factories | `python3 test_shellcode.py` | [core.md](ARCHITECTURE/core.md) | -| Arch/registers/CPU models | `python3 test_cpu_models.py`; `python3 test_riscv.py` | [arch.md](ARCHITECTURE/arch.md) | -| Loaders | `python3 -m unittest test_elf.ELFTest.test_elf_linux_x8664`; `python3 test_uefi.py`; `python3 test_dos.py`; `python3 test_mcu.py` | [loader.md](ARCHITECTURE/loader.md) | -| Memory/paths/structs | `python3 test_pathutils.py && python3 test_struct.py` | [os-base.md](ARCHITECTURE/os-base.md) | -| POSIX syscalls | `python3 test_posix.py` (adds `test_elf.py`, `test_riscv.py`, `test_qltool.py`); then `./test_onlinux.sh` for the CI set | [os-posix.md](ARCHITECTURE/os-posix.md) | -| Kernel proxy | `python3 test_kernel_proxy.py` (Linux host) | [kernel-proxy.md](ARCHITECTURE/kernel-proxy.md) | -| Windows/UEFI/DOS | `python3 test_uefi.py && python3 test_dos.py`; PE suites via `test_pe.bat` on Windows | [os-windows.md](ARCHITECTURE/os-windows.md) | -| MCU/BLOB run loops, peripherals | `python3 test_mcu.py`; `python3 -m unittest test_blob.BlobTest.test_uboot_arm` | [os-baremetal.md](ARCHITECTURE/os-baremetal.md), [hw.md](ARCHITECTURE/hw.md) | -| Debuggers | `python3 test_qdb.py`; `python3 test_debugger.py` | [debugger.md](ARCHITECTURE/debugger.md) | -| Extensions | `python3 test_history.py`; `python3 test_r2.py` with `[RE]` | [extensions.md](ARCHITECTURE/extensions.md) | -| CLI/packaging | `python3 test_qltool.py` (needs the package installed); `python -I tests/test_qltool.py InstalledQltool_Test -v` from the repo root against an installed wheel | [cli.md](ARCHITECTURE/cli.md) | - -CI code checks: `build-ci.yml` runs `tests/test_onlinux.sh` on Ubuntu -(the container branch is dead code, see Roadmap) and -`tests/test_pe.bat` on Windows after `examples/scripts/dllscollector.bat` -(`.github/workflows/build-ci.yml:44-81`); `pythonpublish.yml` runs -`poetry check --lock`, `python -m build`, `twine check --strict`, and the -installed-wheel test on every push and pull request -(`.github/workflows/pythonpublish.yml:3`, `:20-44`). - -Known failing or gated cases on a clean Linux checkout (evidence from -this refresh): `test_blob.BlobTest.test_blob_raw` (missing fixture in the -pinned submodule), `test_elf_ko.ELF_KO_Test.test_demigod_m0hamed_x86` -(needs the unzip step), `test_kernel_proxy…test_ptr_out_writes_back_to_guest_memory` -(test bug, see owner), and `InstalledQltool_Test` when the package is not -installed. Coverage gaps: no unit tests for the hook engine, memory -manager, heap, or calling conventions; Windows and macOS suites need -their hosts. - -Review constraints beyond the shared checks: keep dependency direction -downward; keep guest-derived sizes bounded before host allocation; keep -`unicorn` pinned; new runtime dependencies are blockers (see deviations). - -## Roadmap - -Status: released project in maintenance (1.4.12.dev0). All modules are -`done` except the kernel proxy, which is `in progress (Phase 0)`. -Established implementation milestones exist only for the hybrid kernel -work in `TODO.md` (Phases 0–5, `TODO.md:81-622`); no other milestone IDs -are used in the tree. - -Accepted coding work (evidence-backed, not yet done): - -- Kernel proxy Phase 0 test fix (`tests/test_kernel_proxy.py:449-451`); - Phases 1–5 remain designs (`TODO.md:271-622`). -- Bump `examples/rootfs` so `test_blob_raw` has its fixture - ([os-baremetal.md](ARCHITECTURE/os-baremetal.md)). - -Proposals (from `TODO.md:624-693`, each with an owner and check in the -module docs): remove bare `except:` blocks and `assert` validation; bound -`read_cstring`; structured `map_info` lookup; move the thumb fixup into -the arch layer; implement Windows/UEFI `save/restore`; hook-engine -`isinstance` cleanup; profile-driven guard page; un-skip the ARM and -wchar tests. Cross-cutting gaps: `ChangeLog` stops at 1.4.6 -(`ChangeLog:4`); macOS CI is commented out -(`.github/workflows/build-ci.yml:83-93`); `jexamples/` is unexercised; -`.github/workflows/build-ci.yml:77` reads `matrix.contrainer` (typo) so -the Docker job always takes the native branch. The feature wishlist is -GitHub issue [#333](https://github.com/qilingframework/qiling/issues/333). - -## Development Loop - -Frame → Write → Prove → Review → Gate. Findings return to Write; -uncertainty that changes the plan returns to Frame. - -Use one subagent per role when available, otherwise distinct labeled -passes. Tester and Verifier report findings and never edit; Coder repairs. - -| Role | Stages | Handoff | -| ---- | ------ | ------- | -| Planner | Frame | Goal, observable checks, assumptions, affected files/owners, and plan. | -| Coder | Write | Planned changes or repairs to named findings. | -| Tester | Prove | Commands, results, and behavioral/structural evidence. | -| Verifier | Review + Gate | Evidence-backed findings or verified completion. | - -### The loop - -1. **Frame:** Inspect the request, code, docs, and conventions before - planning. Give the goal and each plan step an observable check. When - using eatmycode, run its Version and Freshness Gate before trusting - architecture; include versions, migration scope, Index/agent-file - changes, and verification commands in architecture plans. Resolve - uncertainty from evidence and record the narrowest supported assumptions. - Only Planner may ask one focused question, when a required decision - cannot be discovered or safely inferred and guessing changes the result. -2. **Write:** Apply Coding Discipline. Make the planned change; for a - repair, address only named findings. Update affected architecture with - changes to its documented contracts. -3. **Prove:** Run relevant tests and structural checks, retaining observable - evidence. For architecture work under eatmycode, apply its Architecture - Verification. Failures and missing, duplicate, or obsolete coverage - become Coder findings. Re-run affected checks after repairs; never send - a red result to Review. -4. **Review:** Apply every Review Check as a separate pass over full affected - files. Use an independent agent or isolated pass for Fit, Dependencies, - and Security when available. Return findings to Coder, then re-prove - and re-review the repairs. -5. **Gate:** Confirm completion only when the Definition of Done passes. - Return unmet criteria to the responsible stage; continue until resolved. - If an external constraint prevents verification, state the missing - evidence and remaining work without claiming completion or readiness. - -Handoffs are automatic. Continue without pauses for plan approval, -permission to continue, or review/reporting ceremonies. Finish with the -harness's normal concise completion handoff. - -### Definition of Done - -- **Correctness:** The goal and named checks pass. Tests cover claimed - behavior; bug fixes have a reproducing regression test. The project - builds and tests from a fresh clone without local-only dependencies. - Owning modules' **How to Test** commands pass with evidence. -- **Review:** Every Review Check ran and its completion threshold passes. -- **Contract:** Docs reflect source and let an agent locate owners, - constraints, and verification commands. When using eatmycode, architecture - satisfies its Output Contract, verification, and version rules. Public - names, signatures, errors, and recovery are intelligible. Breaking - changes, deprecations, dependencies, licenses, and attribution are handled; - commit or PR text, when present, explains why. -- **Scope:** Changed lines serve the goal and follow Coding Discipline; - no debugging remnants, commented-out code, secrets, tokens, or local paths - remain. Test edits follow the inventory and coverage rules below. - -### Iterating without thrashing - -- Each repair pass targets a named finding; nits alone do not trigger one. -- Two no-change passes force Gate re-evaluation. If Done still fails, - return the surviving evidence to Frame. -- Three passes against the same finding return to Frame for a new approach. -- Never widen scope to satisfy a finding. Record coding follow-ups under - **Open Gaps / Roadmap** and keep non-coding work outside architecture. - -## Coding Discipline - -- Implement only the goal. Prefer the simplest approach that passes its - checks; simplify code materially larger than the problem. -- Match local style. Avoid speculative features, flexibility, single-use - abstractions, and checks for impossible conditions. -- Keep edits surgical: no unrelated refactoring, reformatting, or cleanup. - Remove imports, variables, and functions made unused by this change; - leave pre-existing dead code alone unless requested. -- Make success concrete: validation rejects invalid input in a named test; - a regression test fails before a bug fix and passes after; behavior tests - pass before and after a refactor. - -### Before editing tests - -Before any test edit, including during Write, inventory the whole suite: -enumerate every test file and case name, then read in full tests whose -subject, fixtures, or assertions touch the change. Use a subagent for broad -inventory when supported. Plan all additions, changes, merges, and removals -from that evidence, citing `file:line`, before executing the test edits. - -- **Reuse first:** Extend the test owning the behavior or sharing its - setup, fixtures, and subject. Add a function/file only if no existing - owner fits or merging would obscure which case failed. -- **Add only required coverage:** A bug fix needs its regression test; - a capability needs a test of its claimed behavior. Avoid duplicates. -- **Retire only what changed:** Remove tests of deleted behavior and merge - new duplicates, citing surviving coverage. Record unrelated suspected - redundancy under **Open Gaps / Roadmap**. -- **Preserve coverage:** Never delete or weaken tests to turn red green. - Removal needs evidence that behavior is gone or covered elsewhere; - coverage of claimed behavior must not decrease. - -### Project-Specific Deviations - -- Emulation fidelity beats abstraction: syscall, API, and peripheral - implementations mirror the real platform's observable behavior even - when that means repetitive per-OS or per-chip code. Cross-OS - "unification" is a scope increase, not a simplification. -- Coverage is demand-driven by design (see the OS and HW module docs). - Adding an unrequested syscall, Win32 API, or peripheral register is - out of scope; record it under the owning module's **Open Gaps / - Roadmap**. -- `unicorn` is hard-pinned (`pyproject.toml:39`). Changing it, or any - behavior that depends on its version, is a project-wide event and - never an incidental part of another change. -- The kernel proxy must integrate only through `set_syscall` and the fd - table (`TODO.md:65-79`); changes to `load_syscall` or existing handlers - on its behalf are out of scope. - -## Review Checks - -Run every check against every change before confirming a code edit is -complete, even when no commit or merge is requested. Keep checks separate. - -- **Evidence or no finding:** Cite `file:line` for every finding. -- **Repository authority:** Demand only conventions supported by the tree. -- **Full context:** Read affected files, not only hunks; context can expose - unreachable code, unused parameters, or hidden duplication. -- **Code and impact:** Review the change, never the author or how it was made. - -### 1. Style and Naming - -Check indentation and local conventions; leave machine-checkable formatting -to existing formatters/linters and never demand unrelated reformatting. -Mixed indentation is `major`; a consistent new file with the wrong local -indent is `nit`. Compare names with nearby precedents. If the repository -is inconsistent, demand nothing. A local naming mismatch is `nit`; an -inconsistent public name is `major`. - -### 2. Duplication - -Search distinctive constants, errors, fields, and call sequences, beyond -symbol names, for the same job. Cite both sites and a remedy. Cross-layer -duplication is `major`; small local repetition is `nit`. Similar code with -meaningfully different branches is not duplication. - -### 3. Quality - -Require followable control flow, errors handled where they occur, and -proportionate abstractions. Swallowed errors, inappropriate prints, -unexplained magic values, and dead branches are `major`. Remove unrequested -configurability, one-caller wrappers, filler comments, debugging remnants, -and unrelated formatting. Missing tests belong to Prove. - -### 4. Fit - -Read the root architecture and owning module before the diff. Check -language/toolchain constraints, conventions, scope, layering, ownership, -invariants, public-API growth, compatibility, and performance claims against -source. A layering violation or unjustified public API is `major`. -Architectural/public-behavior changes need matching docs in the same change. - -### 5. Dependencies - -Check manifests/imports, maintenance, supply-chain risk, advisories, -install-time behavior, license, transitive cost, and standard-library -alternatives. An unjustified top-level dependency is `major`; a live -advisory or abandoned upstream is `blocker`. Incomplete evidence does not pass. - -### 6. Security - -Check defects and widened exposure: unsafe memory access, unchecked sizes -or offsets, integer overflow, traversal, unsafe deserialization, command -construction, committed secrets, and unbounded untrusted input. Trace input -to impact; without a reachable path there is no finding. A real defect is -`major`; a trust-boundary break is `blocker`. Describe fixes without exploit -steps. - -### Severity and the completion threshold - -| Severity | Effect | -| -------- | ------ | -| `blocker` | Must not confirm completion or merge. | -| `major` | Must be resolved before confirming completion or merging. | -| `nit` | Apply or consciously decline. | -| `info` | Context or a question; no action implied. | - -Confirm completion or merge only with no `blocker` or unresolved `major`. -A check that did not run does not pass; explain evidence-backed -inapplicability. Findings feed Write and Gate directly. - -### Project-Specific Deviations - -- **Security, scope.** Qiling *emulates* untrusted binaries; guest code - doing something hostile inside the sandbox is the product working, not - a finding. Findings target the host boundary: rootfs escape via path - handling (`qiling/os/path.py:239`), unchecked guest-controlled sizes or - offsets reaching host allocations or `struct` unpacking, parser input in - `qiling/loader/` reachable from an untrusted image, and syscalls - forwarded to the host by the kernel proxy - (`qiling/os/posix/kernel_proxy/__init__.py:168-211`; guest-buffer - marshalling at `:237-265`). -- **Dependencies.** `pyproject.toml` is the only manifest. A new - top-level runtime dependency is `blocker` absent an explicit request; - optional integrations belong in an extra (`fuzz`, `RE`). -- **Style and Naming.** No formatter or linter runs in CI; the check enforces - only `.editorconfig` (4-space indent, LF, final newline) and the local - conventions in *Coding Style and Code Design*. - -## Index - -Reading path: run the freshness gate, read the sections above, pick the -owner below from the paths you are touching, then read that module doc -and its listed partners before the code. - -| Module doc | Source paths | Responsibility | Read it when you… | -| ---------- | ------------ | -------------- | ------------------ | -| [core.md](ARCHITECTURE/core.md) | `qiling/__init__.py`, `core.py`, `core_hooks*.py`, `core_struct.py`, `utils.py`, `const.py`, `exception.py`, `log.py`, `host.py`, `profiles/` | `Qiling` facade, composition order, hook engine, component factories, profiles, logging | add a constructor option or hook type, register a new arch/OS/loader name, change save/restore or logging. Partners: every other module | -| [arch.md](ARCHITECTURE/arch.md) | `qiling/arch/`, `qiling/cc/` | CPU layer: Unicorn instance, registers, stack, disassembler, CPU models, calling conventions | add an architecture or CPU model, touch registers/thumb handling, change argument marshalling. Partners: os-base, debugger, os-posix (syscall ABI) | -| [loader.md](ARCHITECTURE/loader.md) | `qiling/loader/` | ELF, PE, PE/UEFI, Mach-O, DOS, MCU firmware, raw blob loading and initial state | change image parsing, entry/exit points, DLL/ld.so resolution, PE cache, MCU peripheral wiring. Partners: os-base, os-posix, os-windows, hw | -| [os-base.md](ARCHITECTURE/os-base.md) | `qiling/os/*.py` | Shared OS services: memory manager/heap, fcall, rootfs paths, fs mapper, fd objects, threads, stats, structs | change memory mapping, path virtualization, API-call protocol, stdio, struct helpers. Partners: all OS personalities, extensions | -| [os-posix.md](ARCHITECTURE/os-posix.md) | `qiling/os/posix/` (except `kernel_proxy/`), `qiling/os/linux/`, `freebsd/`, `macos/`, `qnx/` | Syscall dispatch, ABIs, syscall implementations, Linux threads/futex/procfs, kernel modules, macOS/QNX personalities | add or fix a syscall, change dispatch or fd handling, touch multithread emulation. Partners: os-base, arch, loader, kernel-proxy | -| [kernel-proxy.md](ARCHITECTURE/kernel-proxy.md) | `qiling/os/posix/kernel_proxy/`, `TODO.md` | Opt-in forwarding of chosen syscalls to a real Linux kernel via a helper process | work on the hybrid kernel roadmap, the IPC protocol, proxy fds, or `forward_syscall`. Partners: os-posix, os-base | -| [os-windows.md](ARCHITECTURE/os-windows.md) | `qiling/os/windows/`, `qiling/os/uefi/`, `qiling/os/dos/` | Win32/NT API emulation and kernel objects, UEFI services/protocols, DOS interrupts | add a Win32 API or UEFI protocol, change handles/registry/fibers/threads, touch DOS interrupts. Partners: loader, os-base, arch | -| [os-baremetal.md](ARCHITECTURE/os-baremetal.md) | `qiling/os/mcu/`, `qiling/os/blob/`, `qiling/extensions/multitask.py` | MCU and raw-blob run loops, cooperative multitasking over Unicorn | change stepping/fast mode, interrupt delivery timing, blob execution. Partners: hw, arch, loader | -| [hw.md](ARCHITECTURE/hw.md) | `qiling/hw/`, `qiling/extensions/mcu/` | MMIO peripherals, hardware manager, chip `env` maps | add a peripheral or chip, change MMIO routing or peripheral hooks. Partners: os-baremetal, loader, os-base | -| [debugger.md](ARCHITECTURE/debugger.md) | `qiling/debugger/` | GDB remote-serial server and Qdb (stepping, branch prediction, record/replay) | change RSP handling, target XML, Qdb commands or per-arch support. Partners: core, arch | -| [extensions.md](ARCHITECTURE/extensions.md) | `qiling/extensions/` (except `multitask.py`, `mcu/`) | AFL fuzzing, coverage/trace writers, heap sanitizer, r2/IDA integration, pipes, reports, SDK stub generator | add a coverage/trace format, fuzzing harness support, sanitizer, or analysis integration. Partners: core, os-base, cli | -| [cli.md](ARCHITECTURE/cli.md) | `qltool`, `qiling/cli.py`, `qltui.py` | `qltool` subcommands/flags and the TUI; packaging smoke test | add or change a CLI flag, TUI prompt, or console-script behavior. Partners: core, debugger, extensions | +`Qiling.__init__` composes arch → hooks/packing → logging/profile → loader → +memory → OS → optional hardware, then runs the loader. `run()` applies +patches and delegates to the OS loop; Unicorn hooks enter Python dispatch. +Arch owns CPU/register state, memory owns mappings, loaders own image state, +and OS personalities own process/API state. Factory names are compatibility +contracts ([core](qiling/core.py), [factories](qiling/utils.py)). + +Guest bytes/pointers cross host boundaries through loaders, memory, files +and sockets. Rootfs handling is not complete +host isolation; explicit mappings expose host objects. Snapshot files use +pickle and require trusted producers. The opt-in kernel proxy executes real +host syscalls. Follow the owners below for these contracts. Cooperative +scheduling and partial snapshots do not provide complete machine recovery. + +## Code Conventions + +- **Required:** [.editorconfig](.editorconfig) sets Python UTF-8, LF, four + spaces, final newline and trailing-space removal. No formatter, linter or + type-checker gate is configured in the manifest/CI. +- **Observed:** `Ql*` classes, snake_case helpers, uppercase constants; + legacy camelCase APIs and wildcard imports remain. Follow nearby code, + preserve public names, use `ql.log` and `QlError*` where established; + annotations are mixed ([core](qiling/core.py), [errors](qiling/exception.py)). +- **Preserved project rules:** favor platform fidelity, keep API/peripheral + additions demand-driven, and treat Unicorn upgrades as project-wide work. + New top-level runtime dependencies require an explicit request; optional + integrations use extras. These strengthen shared review (prior project + architecture; current [manifest](pyproject.toml)). Keep dependencies toward + underlying services; read [core](ARCHITECTURE/modules/core.md) before + changing cross-layer imports for the existing exceptions. +- `AGENT.md` and `CLAUDE.md` alias this page; asset/fixture edit constraints + live with owners. + +## Verification + +| Change/check | Command and working directory | Prerequisites / pass evidence | +| --- | --- | --- | +| Setup | `python3 -m venv .venv`; activate it; `python -m pip install -e .` — root | Python in declared range; engine wheels/build prerequisites. | +| Local CLI | `python qltool --help` — root | Installed dependencies; help exits zero. | +| Shared primitives | `python -m unittest test_pathutils test_struct` — `tests/` | Passing assertions; no full emulation claim. | +| Runtime changes | Owner commands below; Linux aggregate `./test_onlinux.sh` — `tests/` | [Rootfs submodule](.gitmodules), test-specific libraries/hosts; aggregate stops on first failure. | +| Build / metadata | [CLI and build checks](ARCHITECTURE/modules/cli-build.md#verification) for packaging/config changes | Mirrors [package CI](.github/workflows/pythonpublish.yml); lint/type checks unavailable. | + +## Task Index + +| Source paths / task trigger | Responsibility | Read next | +| --- | --- | --- | +| `qiling/{core*,utils,const,exception,host,log,__init__}.py`, `arch/`, `cc/`, `loader/`, `profiles/`; composition, CPU, images | Runtime and loading | [Runtime routes](ARCHITECTURE/indexes/runtime.md) | +| `qiling/os/` shared files, `posix/` except `kernel_proxy/`, `linux/`, `freebsd/`, `macos/`, `qnx/`, `windows/`, `uefi/`, `dos/`; OS behavior | Services and personalities | [OS routes](ARCHITECTURE/indexes/operating-systems.md) | +| `qiling/os/{mcu,blob}/`, `hw/`, `extensions/{mcu/,multitask.py}`; firmware/devices | Firmware and MMIO | [Firmware routes](ARCHITECTURE/indexes/firmware.md) | +| `qiling/os/posix/kernel_proxy/`, hybrid-kernel sections of `TODO.md`, `tests/test_kernel_proxy.py`; forwarding | Host-kernel integration | [Kernel proxy](ARCHITECTURE/modules/kernel-proxy.md) | +| `qiling/debugger/`, `tests/test_*debugger.py`, `test_qdb.py`, `qdb_scripts/`; debugging | GDB and Qdb | [Debugger](ARCHITECTURE/modules/debugger.md) | +| Other `qiling/extensions/`, `examples/fuzzing/`, `tests/test_{history,r2}.py`; integrations | Instrumentation | [Extensions](ARCHITECTURE/modules/extensions.md) | +| `qiling/cli.py`, `qltool`, `qltui.py`, manifests/lock, `Dockerfile`, `.github/`, test drivers, `.gitmodules`; tooling | CLI and development checks | [CLI and build](ARCHITECTURE/modules/cli-build.md) | diff --git a/ARCHITECTURE/AGENT_RULES.md b/ARCHITECTURE/AGENT_RULES.md new file mode 100644 index 000000000..ab0bd43f4 --- /dev/null +++ b/ARCHITECTURE/AGENT_RULES.md @@ -0,0 +1,187 @@ +--- +eatmycode_version: "2.0.0" +--- + +# Agent Rules + +Owner: [Project architecture](../ARCHITECTURE.md) + +Read when: before planning code changes or reviewing code. + +## Development Loop + +Frame → Write → Prove → Review → Gate. Findings return to Write; +uncertainty that changes the plan returns to Frame. + +Use one subagent per role when available, otherwise distinct labeled +passes. Tester and Verifier report findings and never edit; Coder repairs. + +| Role | Stages | Handoff | +| ---- | ------ | ------- | +| Planner | Frame | Goal, observable checks, assumptions, affected files/owners, and plan. | +| Coder | Write | Planned changes or repairs to named findings. | +| Tester | Prove | Commands, results, and behavioral/structural evidence. | +| Verifier | Review + Gate | Evidence-backed findings or verified completion. | + +### The loop + +1. **Frame:** Inspect the request, code, docs, and conventions before + planning. Give the goal and each plan step an observable check. When + using eatmycode, run its Version and Freshness Gate before trusting + architecture; include versions, migration scope, routing/agent-file + changes, and verification commands in architecture plans. Resolve + uncertainty from evidence and record the narrowest supported assumptions. + Only Planner may ask one focused question, when a required decision + cannot be discovered or safely inferred and guessing changes the result. +2. **Write:** Apply Coding Discipline. Make the planned change; for a + repair, address only named findings. Update affected architecture with + changes to its documented contracts. +3. **Prove:** Run relevant tests and structural checks, retaining observable + evidence. For architecture work under eatmycode, apply its Architecture + Verification. Failures and missing, duplicate, or obsolete coverage + become Coder findings. Re-run affected checks after repairs; never send + a red result to Review. +4. **Review:** Apply every Review Check as a separate pass over full affected + files. Use an independent agent or isolated pass for Fit, Dependencies, + and Security when available. Return findings to Coder, then re-prove + and re-review the repairs. +5. **Gate:** Confirm completion only when the Definition of Done passes. + Return unmet criteria to the responsible stage; continue until resolved. + If an external constraint prevents verification, state the missing + evidence and remaining work without claiming completion or readiness. + +Handoffs are automatic. Continue without pauses for plan approval, +permission to continue, or review/reporting ceremonies. Finish with the +harness's normal concise completion handoff. + +### Definition of Done + +- **Correctness:** The goal and named checks pass. Tests cover claimed + behavior; bug fixes have a reproducing regression test. The project + builds and tests from a fresh clone without local-only dependencies. + Owning modules' **Verification** commands pass with evidence. +- **Review:** Every Review Check ran and its completion threshold passes. +- **Contract:** Docs reflect source and let an agent locate owners, + constraints, and verification commands. When using eatmycode, architecture + satisfies its Output Contract, verification, and version rules. Public + names, signatures, errors, and recovery are intelligible. Breaking + changes, deprecations, dependencies, licenses, and attribution are handled; + commit or PR text, when present, explains why. +- **Scope:** Changed lines serve the goal and follow Coding Discipline; + no debugging remnants, commented-out code, secrets, tokens, or local paths + remain. Test edits follow the inventory and coverage rules below. + +### Iterating without thrashing + +- Each repair pass targets a named finding; nits alone do not trigger one. +- Two no-change passes force Gate re-evaluation. If Done still fails, + return the surviving evidence to Frame. +- Three passes against the same finding return to Frame for a new approach. +- Never widen scope to satisfy a finding. Record coding follow-ups under + **Known Gaps** and keep non-coding work outside architecture. + +## Coding Discipline + +- Implement only the goal. Prefer the simplest approach that passes its + checks; simplify code materially larger than the problem. +- Match local style. Avoid speculative features, flexibility, single-use + abstractions, and checks for impossible conditions. +- Keep edits surgical: no unrelated refactoring, reformatting, or cleanup. + Remove imports, variables, and functions made unused by this change; + leave pre-existing dead code alone unless requested. +- Make success concrete: validation rejects invalid input in a named test; + a regression test fails before a bug fix and passes after; behavior tests + pass before and after a refactor. + +### Before editing tests + +Before any test edit, including during Write, inventory the whole suite +with discovery tools; keep the full file/case listing outside model context. +Load matching inventory entries and read in full tests whose subject, +fixtures, or assertions touch the change. Use a subagent for broad inventory +when supported. Plan all additions, changes, merges, and removals from that +evidence, citing `file:line`, before executing the test edits. + +- **Reuse first:** Extend the test owning the behavior or sharing its + setup, fixtures, and subject. Add a function/file only if no existing + owner fits or merging would obscure which case failed. +- **Add only required coverage:** A bug fix needs its regression test; + a capability needs a test of its claimed behavior. Avoid duplicates. +- **Retire only what changed:** Remove tests of deleted behavior and merge + new duplicates, citing surviving coverage. Record unrelated suspected + redundancy under **Known Gaps**. +- **Preserve coverage:** Never delete or weaken tests to turn red green. + Removal needs evidence that behavior is gone or covered elsewhere; + coverage of claimed behavior must not decrease. + +## Review Checks + +Run every check against every change before confirming a code edit is +complete, even when no commit or merge is requested. Keep checks separate. + +- **Evidence or no finding:** Cite `file:line` for every finding. +- **Repository authority:** Demand only conventions supported by the tree. +- **Full context:** Read affected files, not only hunks; context can expose + unreachable code, unused parameters, or hidden duplication. +- **Code and impact:** Review the change, never the author or how it was made. + +### 1. Style and Naming + +Check indentation and local conventions; leave machine-checkable formatting +to existing formatters/linters and never demand unrelated reformatting. +Mixed indentation is `major`; a consistent new file with the wrong local +indent is `nit`. Compare names with nearby precedents. If the repository +is inconsistent, demand nothing. A local naming mismatch is `nit`; an +inconsistent public name is `major`. + +### 2. Duplication + +Search distinctive constants, errors, fields, and call sequences, beyond +symbol names, for the same job. Cite both sites and a remedy. Cross-layer +duplication is `major`; small local repetition is `nit`. Similar code with +meaningfully different branches is not duplication. + +### 3. Quality + +Require followable control flow, errors handled where they occur, and +proportionate abstractions. Swallowed errors, inappropriate prints, +unexplained magic values, and dead branches are `major`. Remove unrequested +configurability, one-caller wrappers, filler comments, debugging remnants, +and unrelated formatting. Missing tests belong to Prove. + +### 4. Fit + +Follow the root Read First instructions and read the owning module before +the diff. Check language/toolchain constraints, conventions, scope, layering, +ownership, invariants, public-API growth, compatibility, and performance +claims against source. A layering violation or unjustified public API is `major`. +Architectural/public-behavior changes need matching docs in the same change. + +### 5. Dependencies + +Check manifests/imports, maintenance, supply-chain risk, advisories, +install-time behavior, license, transitive cost, and standard-library +alternatives. An unjustified top-level dependency is `major`; a live +advisory or abandoned upstream is `blocker`. Incomplete evidence does not pass. + +### 6. Security + +Check defects and widened exposure: unsafe memory access, unchecked sizes +or offsets, integer overflow, traversal, unsafe deserialization, command +construction, committed secrets, and unbounded untrusted input. Trace input +to impact; without a reachable path there is no finding. A real defect is +`major`; a trust-boundary break is `blocker`. Describe fixes without exploit +steps. + +### Severity and the completion threshold + +| Severity | Effect | +| -------- | ------ | +| `blocker` | Must not confirm completion or merge. | +| `major` | Must be resolved before confirming completion or merging. | +| `nit` | Apply or consciously decline. | +| `info` | Context or a question; no action implied. | + +Confirm completion or merge only with no `blocker` or unresolved `major`. +A check that did not run does not pass; explain evidence-backed +inapplicability. Findings feed Write and Gate directly. diff --git a/ARCHITECTURE/arch.md b/ARCHITECTURE/arch.md deleted file mode 100644 index d729d6804..000000000 --- a/ARCHITECTURE/arch.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -eatmycode_version: "1.2.0" ---- - -# Arch — CPU architecture layer - -## Goal - -Own everything CPU-specific: the Unicorn `Uc` instance, register access, -stack primitives, disassembler/assembler, CPU models, and per-arch calling -conventions. This is the bottom layer: every other module reads `ql.arch`; -arch depends only on Unicorn/Capstone/Keystone (with the documented -exceptions below). No roadmap milestone applies; maturity-based status. - -## Status - -`done` — all ten architectures are exercised by the CI suites; the CPU -model enums are checked against Unicorn's constants by -`tests/test_cpu_models.py` (observed: `Ran 7 tests … OK`). - -## Code Structure - -| File | Role | -| ---- | ---- | -| `qiling/arch/arch.py` | Abstract base `QlArch`: owns `uc`, `regs`, stack push/pop, save/restore, disassembler/assembler | -| `qiling/arch/x86.py`, `x86_utils.py`, `x86_const.py`, `msr.py` | `QlArchIntel` base + `QlArchA8086`/`QlArchX86`/`QlArchX8664`; GDT/segment setup; MSRs | -| `qiling/arch/arm.py`, `arm_utils.py`, `arm_const.py`, `cpr.py` | ARM: thumb handling, coprocessor registers | -| `qiling/arch/arm64.py`, `arm64_const.py`, `cpr64.py` | AArch64 | -| `qiling/arch/cortex_m.py`, `cortex_m_const.py` | Cortex-M on top of ARM: `QlInterruptContext`, NVIC-style exception entry/exit for MCU mode; uses `MultiTaskUnicorn` | -| `qiling/arch/mips.py`, `riscv.py`, `riscv64.py`, `ppc.py` (+ `*_const.py`) | Remaining architectures | -| `qiling/arch/register.py` | `QlRegisterManager`: attribute-style register read/write | -| `qiling/arch/models.py` | CPU model enums (`X86_CPU_MODEL` … `RISCV64_CPU_MODEL`) | -| `qiling/arch/utils.py` | `QlArchUtils` (disassembly output for verbose modes) and the `assembler()` factory | -| `qiling/cc/__init__.py`, `intel.py`, `arm.py`, `mips.py`, `ppc.py`, `riscv.py` | Calling conventions (argument/return marshalling) consumed by `qiling/os/fcall.py` | - -## Language and Conventions - -Python; root rules apply. Local patterns: - -- One class per architecture named `QlArch` so `select_arch` - can derive it (`qiling/utils.py:376-406`). -- Register tables live in `*_const.py` as name→Unicorn-constant maps and are - handed to `QlRegisterManager` (`qiling/arch/register.py:16`). -- Calling conventions are small classes named after the ABI (`cdecl`, - `stdcall`, `ms64`, `amd64`, `macosx64`, `aarch64`, `aarch32`, `mipso32`; - `qiling/cc/intel.py:61-95`, `qiling/cc/arm.py:35-40`, - `qiling/cc/mips.py:9`), all deriving from `QlCommonBaseCC` - (`qiling/cc/__init__.py:110`). -- `TODO.md:638-644` records that GDT/segment validation in - `qiling/arch/x86_utils.py` uses `assert`; treat that as observed, not a - convention to copy. - -## Design and Invariants - -- `QlArch` declares `uc` as an abstract property that each concrete arch - builds lazily as a `cached_property` (`qiling/arch/x86.py:27`); the base - (`qiling/arch/arch.py:32-34`) exposes `regs` (`:42`), - `stack_push/stack_pop` (`:52`/`:66`), `save/restore` via `UcContext` - (`:108`/`:112`), `disassembler` (`:117`), and `assembler` (`:125`). - Everything above arch must go through these. -- `ql.uc` is a proxy to `arch.uc` (`qiling/core.py:479`); there is exactly - one Unicorn instance per `Qiling` (the multi-Unicorn threading idea in - `TODO.md:462-488` is a proposal only). -- **Layering exceptions**: `qiling/arch/cortex_m.py:22` imports - `MultiTaskUnicorn` from `qiling/extensions/multitask.py`; - `qiling/arch/utils.py:94` lazily imports the r2 extension for symbol - names; `qiling/arch/x86_utils.py:10` imports `QlMemoryManager` from OS - base for the GDT manager's constructor annotation. Do not add further - upward imports; see [os-baremetal.md](os-baremetal.md) for the multitask - contract. -- CPU models come in through the `cputype` kwarg; `select_arch` forwards - the value unvalidated (`qiling/utils.py:377`) and the arch class applies - it with `ctl_set_cpu_model` (`qiling/arch/arm.py:47`); a model belongs - to exactly one enum in `qiling/arch/models.py`. -- Endianness and thumb are constructor inputs for ARM/MIPS only - (`qiling/utils.py:379-386`). - -## Key Types and Entry Points - -- `qiling/arch/arch.py:22` - `QlArch(ABC)` - base class; see properties - above. -- `qiling/arch/register.py:11` - `QlRegisterManager` - `ql.arch.regs.rax` - style access (`__getattr__` `:35`, `__setattr__` `:44`), plus - `read/write` by name or Unicorn id (`:53`/`:62`). -- `qiling/arch/x86.py:22,53,79,111` - `QlArchIntel` / `QlArchA8086` / - `QlArchX86` / `QlArchX8664`. -- `qiling/arch/cortex_m.py:67` - `QlArchCORTEX_M(QlArchARM)` - - `interrupt_handler` (`:146`) consults `ql.hw.nvic` and enters the handler - inside `QlInterruptContext` (`:25`). -- `qiling/arch/utils.py:106` - `assembler(arch, endianness, is_thumb)` - - Keystone factory used by `qltool code --format asm`. -- `qiling/cc/__init__.py:9` - `QlCC` - abstract calling convention - (`getRawParam`, `setReturnValue`, …); `QlCommonBaseCC` (`:110`). -- `qiling/utils.py:376` - `select_arch(archtype, cputype, endian, thumb)` - - the only construction path (`qiling/core.py:154`). - -## Interactions - -- Constructed first by [core.md](core.md); `QlCoreStructs`/`QlCoreHooks` - are initialized from `arch.endian`/`arch.bits`/`arch.uc` - (`qiling/core.py:157-158`). -- [loader.md](loader.md) and the OS layers set initial register/stack state - through `arch.regs` and the stack primitives. -- `qiling/cc/` is consumed by `QlFunctionCall` ([os-base.md](os-base.md)) - and by the Windows fcall selector (`qiling/os/windows/windows.py:41-65`). -- POSIX syscall ABIs are a separate table in - `qiling/os/posix/syscall/abi/` ([os-posix.md](os-posix.md)), not here. -- [debugger.md](debugger.md) reads/writes registers through this layer and - ships per-arch GDB target XML. -- [hw.md](hw.md) NVIC peripherals call `arch.interrupt_handler` - (`qiling/hw/intc/cm_nvic.py:55`, `:127`). - -## How to Test - -```sh -cd tests && python3 test_cpu_models.py # pass = "Ran 7 tests … OK", exit 0 -``` - -- Broader coverage comes from `tests/test_shellcode.py` (five archs) and the - per-OS suites; RISC-V has `tests/test_riscv.py` (`Ran 4 tests … OK`). -- There is no unit test for `qiling/cc/`; it is proven through Windows API - calls (Windows host) and UEFI (`tests/test_uefi.py`). - -## Review and Refactor Guide - -- **New architecture**: add `qiling/arch/.py` with `QlArch`, - a `*_const.py` register table, a `qiling/cc/` convention, a - `qiling/os/posix/syscall/abi/` ABI, a GDB XML directory, and the enum in - `qiling/const.py:15`; then extend `select_arch`. -- **Register changes** affect `qiling/debugger/gdb/xml//` and - `qiling/debugger/qdb/arch/`; run `tests/test_debugger.py` and - `tests/test_qdb.py`. -- **Do not** put OS-specific state (segment selectors for Windows, TLS) in - arch classes; the OS layer owns that (`qiling/os/windows/windows.py:132`). -- Improvement candidate (proposal): move the thumb-bit fixup out of - `Qiling.emu_start` (`qiling/core.py:759`) into `QlArchARM` once Unicorn - reflects thumb mode in `cpsr` at init. Success check: - `tests/test_shellcode.py::test_linux_arm_thumb` and `tests/test_qdb.py` - stay green. - -## Open Gaps / Roadmap - -- PowerPC has no OS-level test suite beyond CPU model selection; RISC-V has - four Linux tests. -- 8086 GDB stop replies use the wrong register names (FIXME at - `qiling/debugger/gdb/gdb.py:242`). -- `TODO.md:655-657` notes the 32-bit GDT data segment is built with - `QL_X86_A_PRIV_0` (`qiling/arch/x86_utils.py:167`); the 64-bit manager - already uses ring 3 (`:200`, `:215`). diff --git a/ARCHITECTURE/cli.md b/ARCHITECTURE/cli.md deleted file mode 100644 index 1791f2810..000000000 --- a/ARCHITECTURE/cli.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -eatmycode_version: "1.2.0" ---- - -# CLI — qltool and qltui - -## Goal - -Give users a no-code way to run emulations: `qltool run` executes a binary -against a rootfs, `qltool code` runs shellcode (hex/asm/bin), `qltool -examples` prints usage samples, and `qltool qltui` launches an interactive -TUI that gathers the same options. Owns argument parsing and the mapping -from flags to `Qiling` kwargs; it must not implement emulation behavior. -No roadmap milestone applies; maturity-based status. - -## Status - -`done` — `tests/test_qltool.py` (observed: `Ran 8 tests … OK` once the -package is installed) shells out to `qltool` for run/code subcommands, -coverage, JSON, and log filtering; `InstalledQltool_Test` checks the -installed console script, shellcode exit status, bundled profiles, and TUI -import outside the checkout. - -## Code Structure - -| File | Role | -| ---- | ---- | -| `qltool` | Checkout launcher: `from qiling.cli import run` | -| `qiling/cli.py` | Argparse CLI; installed as the `qltool` console script (`pyproject.toml:33-34`); builds kwargs and drives `Qiling` | -| `qltui.py` | questionary/pyfx/termcolor TUI; collects options and returns them to `qltool` | - -## Language and Conventions - -Python; root rules apply. Enum-valued flags use `__make_enum_arg` -argparse actions mapping lowercase names to `QL_ARCH`/`QL_OS`/`QL_ENDIAN`/ -`QL_VERBOSE` (`qiling/cli.py:59-75`). Errors surface as argparse errors or -`Qiling` exceptions; the process exits with `ql.os.exit_code` -(`qiling/cli.py:321`). `qltui.py` is the only module that imports -`questionary`, `pyfx`, and `termcolor` (`pyproject.toml:48-50`). - -## Design and Invariants - -- Subcommands: `run` (`-f`, `--rootfs`, `--args …`), `code` (`-f`/`-i`, - `--arch`, `--os`, `--endian`, `--thumb`, `--format asm|hex|bin`), - `examples`, `qltui` (`qiling/cli.py:196-217`); common flags cover - verbosity, `--env` (pickled dict), `--gdb`, `--qdb`, `--rr`, - `--profile`, `--filter`, `--log-file`, `--log-plain`, `--no-console`, - `--root`, `--debug-stop`, `--multithread`, `--timeout`, `--coverage-file`, - `--coverage-format`, `--json`, `--libcache` (`:225-242`). -- `handle_run`/`handle_code` return the kwargs dict; `Qiling(**ql_args)` - at `qiling/cli.py:276` is the single construction point, followed by - optional Qdb (`:279`), gdbserver (`:285`), coverage-wrapped `ql.run()` - (`:306-310`), JSON report (`:312`), and exit (`:321`). -- `code --format asm` assembles with Keystone via - `qiling.arch.utils.assembler` (`:104`). -- The installed script and the checkout launcher must behave identically; - `InstalledQltool_Test` runs with an empty `PYTHONPATH` from a temp - directory to prove profiles ship in the wheel (`tests/test_qltool.py:55-72`). - -## Key Types and Entry Points - -- `qiling/cli.py:189` - `run()` - argparse setup and dispatch. -- `qiling/cli.py:129` - `handle_run(options)` - builds - `{'argv': [file]+args, 'rootfs': …}`. -- `qiling/cli.py:78` - `handle_code(options)` - reads hex/asm/bin shellcode. -- `qiling/cli.py:59` - `__make_enum_arg(enum_rmap, aliases)` - argparse - action factory. -- `qltui.py` - TUI entry invoked by the `qltui` subcommand. - -## Interactions - -- Thin client of [core.md](core.md): constructs `Qiling` and calls `run()`. -- Attaches [debugger.md](debugger.md) via `--gdb [HOST:PORT]` / - `--qdb [--rr]`. -- Uses [extensions.md](extensions.md) for coverage (`cov_utils.factory`) - and the JSON report. -- Packaging: `pyproject.toml:33-34` defines the console script; the PyPI - workflow runs `InstalledQltool_Test` against the built wheel - (`.github/workflows/pythonpublish.yml:41-44`). - -## How to Test - -```sh -python3 -m pip install -e . -cd tests && python3 test_qltool.py # pass = "Ran 8 tests … OK", exit 0 -``` - -- `Qltool_Test` uses the checkout launcher `../qltool`; - `InstalledQltool_Test` needs the console script on the interpreter's - scripts path (hence the install step). Without it the two installed - cases error with `FileNotFoundError: …/bin/qltool`. -- Manual smoke test from the repository root (pass = prints - `Hello, World!`): - - ```sh - ./qltool run -f examples/rootfs/x8664_linux/bin/x8664_hello --rootfs examples/rootfs/x8664_linux - ``` - -## Review and Refactor Guide - -- **New flag**: add it to the common or subcommand parser, thread it into - `ql_args` or the post-construction block, mirror it in `qltui.py`, and - add a `Qltool_Test` case. -- **Do not** add emulation logic here; new behavior belongs to the owning - module and is only surfaced by a flag. -- Keep `qltool` (checkout) a two-line launcher so installed and checkout - behavior cannot diverge. - -## Open Gaps / Roadmap - -- `qltui.py` has only an import smoke test; its interactive flows are not - covered. -- Complex setups (fs mappers, custom hooks) still require the Python API. diff --git a/ARCHITECTURE/core.md b/ARCHITECTURE/core.md deleted file mode 100644 index ffa4f5d66..000000000 --- a/ARCHITECTURE/core.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -eatmycode_version: "1.2.0" ---- - -# Core — the Qiling facade and plumbing - -## Goal - -Own the public API and object lifecycle of an emulation: the `Qiling` class -composes arch, loader, memory, OS, and (bare-metal only) hardware components, -and exposes hooks, memory/register access, patching, save/restore, and the -component factories that resolve names to classes. It must not implement any -CPU, format, OS, or peripheral behavior itself. No roadmap milestone applies; -this is released infrastructure with maturity-based status. - -## Status - -`done` — exercised by every suite; the **How to Test** command boots -shellcode end-to-end through `Qiling.__init__` → `run()` → `emu_start` -(observed: `Ran 6 tests … OK`). - -## Code Structure - -| File | Role | -| ---- | ---- | -| `qiling/__init__.py` | Exports `Qiling`; `__version__` from package metadata | -| `qiling/core.py` | `Qiling`: constructor/composition root, `run`, `emu_start/stop`, `save/restore`, `patch`, properties, stop guard | -| `qiling/core_hooks.py` | `QlCoreHooks` mixin: wraps Unicorn hooks, dispatches to `Hook` lists, honors `QL_HOOK_BLOCK` | -| `qiling/core_hooks_types.py` | `Hook`, `HookAddr`, `HookIntr`, `HookRet` records | -| `qiling/core_struct.py` | `QlCoreStructs` mixin: endian/bit-width-aware `pack*/unpack*` helpers | -| `qiling/utils.py` | Name→class factories (`select_arch/loader/os/component/debugger`), binary sniffing, profile loading | -| `qiling/const.py` | Enums `QL_ARCH`, `QL_OS`, `QL_VERBOSE`, `QL_INTERCEPT`, `QL_STOP`, `QL_STATE`; groupings `QL_OS_POSIX`, `QL_OS_BAREMETAL`; hook flags | -| `qiling/exception.py` | `QlErrorBase` and its subclasses | -| `qiling/host.py` | `QlHost`: describes the *hosting* platform for pass-through decisions | -| `qiling/log.py` | Logger setup, colored/plain formatters, regex filter behind `Qiling.filter` | -| `qiling/profiles/*.ql` | Default per-OS INI profiles merged with user overrides | - -## Language and Conventions - -Python only; root toolchain and style rules apply (see -[ARCHITECTURE.md](../ARCHITECTURE.md#coding-style-and-code-design)). Local -patterns to follow: - -- Circular-import avoidance with `TYPE_CHECKING` guards and string - annotations (`qiling/core.py:1-33`, `qiling/log.py:19-20`). -- Components are reached through read-only properties (`mem`, `arch`, - `loader`, `os`, `hw`, `log`; `qiling/core.py:203-244`); setters exist only - for `verbose`, `debugger`, `filter`, `debug_stop`. -- Public hook methods are fully typed with callback `Protocol`s - (`qiling/core_hooks.py:55-131`). -- Errors are `QlErrorBase` subclasses (`qiling/exception.py:9`); constructor - failures raise `QlErrorFileNotFound`, `QlErrorArch`, `QlErrorOsType` - (`qiling/core.py:104`, `:143-149`). - -## Design and Invariants - -- **Composition order is fixed** (`qiling/core.py:154-197`): arch → - `QlCoreStructs`/`QlCoreHooks` init → logger → profile → loader → memory → - OS → hw (bare-metal only) → `loader.run()` → stop guard. Later components - read earlier ones during their own `__init__` (e.g. the OS reads - `ql.profile` and `ql.mem`), so reordering breaks them. -- **Core never imports concrete subclasses.** All resolution goes through - the dynamic-import factories in `qiling/utils.py` (`select_arch` `:376`, - `select_loader` `:297`, `select_os` `:409`, `select_component` `:323`, - `select_debugger` `:332`), which derive module and class names from the - enum names. Adding an arch/OS means adding a module whose class name - matches that derivation. -- **Hook dispatch protocol**: `QlCoreHooks` registers one Unicorn hook per - type and fans out to Python `Hook` lists; a callback returning an int with - `QL_HOOK_BLOCK` set (`qiling/const.py:77`) stops remaining hooks - (`qiling/core_hooks.py:186`, `:209`). Address hooks are keyed per address - (`hook_address`, `:550`). `begin=1, end=0` means "whole address space". -- **Exceptions raised inside hooks** are captured by the hook wrapper - (`qiling/core_hooks.py:141-144`) into `ql._internal_exception`, which - `emu_start` resets beforehand and re-raises after `uc.emu_start` returns - (`qiling/core.py:763`, `:773-774`), because Unicorn cannot propagate - Python exceptions through its C callbacks. -- **Emulation state** is tracked in `QL_STATE` (`qiling/const.py:67`) around - `emu_start` (`qiling/core.py:765-770`); `QlOs.call` refuses to move `pc` - once stopped to work around a Unicorn bug (`qiling/os/os.py:215-221`). -- **Stop guard**: when `stop=QL_STOP.*` is requested, a trap page is mapped - at or above `0x9000000` (`qiling/core.py:525`) and the loader's - `skip_exit_check` decides whether the trap is written (`:546-558`). -- **Thumb workaround**: `emu_start` forces the low bit of `begin` when the - arch was initialized in thumb mode (`qiling/core.py:759`); the FIXME there - explains why this cannot live in the arch layer today. -- **Save/restore** snapshots are per-component dicts (`qiling/core.py:609`, - `:658`); each component implements its own `save()/restore()`. - -## Key Types and Entry Points - -- `qiling/core.py:35` - `Qiling(QlCoreHooks, QlCoreStructs)` - facade; - constructor kwargs at `:36-58` are the public construction contract - (`argv`, `rootfs`, `env`, `code`, `ostype`, `archtype`, `cputype`, - `verbose`, `profile`, `multithread`, `stop`, `endian`, `thumb`, - `libcache`, …). -- `qiling/core.py:561` - `Qiling.run(begin, end, timeout, count)` - - instantiates the debugger if configured, applies binary patches, writes - the exit trap, calls `os.run()`, then `debugger.run()`. -- `qiling/core.py:743` - `Qiling.emu_start(begin, end, timeout, count)` - - thin wrapper over `uc.emu_start`; manages thumb bit, `QL_STATE`, and - exception re-raise. -- `qiling/core.py:609` / `:658` - `save()` / `restore()` - snapshot - regs/mem/hw/fd/os/loader per component; optional pickle via `snapshot=`. -- `qiling/core.py:594` - `patch(offset, data, target=None)` - queue a binary - or library patch applied by `do_bin_patch`/`do_lib_patch` (`:503`, `:509`). -- `qiling/core_hooks.py:150` - `QlCoreHooks` - `hook_code` (`:400`), - `hook_block` (`:422`), `hook_address` (`:550`), `hook_intno` (`:575`), - `hook_mem_read/write` (`:592`/`:610`), `hook_insn` (`:646`), - `hook_del` (`:686`). -- `qiling/utils.py:278` - `ql_guess_emu_env(path)` - sniffs arch/OS/endian - from path name, ELF, Mach-O, or PE headers when not given. -- `qiling/utils.py:419` - `profile_setup(ostype, user_config)` - YAML for - MCU, else `ConfigParser` over `qiling/profiles/.ql` plus user overrides - (path or dict); `getint` accepts any base. -- `qiling/log.py:164` - `setup_logger(ql, logdevs, plain, override)` - builds - the per-instance logger; `RegexFilter` (`:96`) implements `ql.filter`. - -## Interactions - -- Instantiates [arch.md](arch.md), [loader.md](loader.md), - [os-base.md](os-base.md) (memory then OS), and [hw.md](hw.md) (bare-metal - only) in the order above. -- Lazily instantiates [debugger.md](debugger.md) inside `run()` via - `select_debugger`. -- [extensions.md](extensions.md), [cli.md](cli.md), and - [kernel-proxy.md](kernel-proxy.md) consume only this public API. -- The OS layers register their syscall/API entry hooks through - `QlCoreHooks` (`hook_intno`/`hook_insn`/`hook_code`/`hook_intr`, see - [os-posix.md](os-posix.md), [os-windows.md](os-windows.md)). - -## How to Test - -```sh -cd tests && python3 test_shellcode.py # pass = "Ran 6 tests … OK", exit 0 -``` - -- Covers `Qiling(code=…)` construction and `run()` for x86/x86-64/ARM/ - ARM64/MIPS Linux shellcode (`tests/test_shellcode.py:90`). -- Hook semantics and save/restore are exercised indirectly by - `tests/test_elf.py` and `tests/test_mcu.py` (`test_mcu_snapshot_stm32f411`, - `tests/test_mcu.py:29`). There is no dedicated unit test for - `QlCoreHooks`; see Open Gaps. - -## Review and Refactor Guide - -- **Adding a constructor option**: extend `Qiling.__init__` - (`qiling/core.py:36`) and mirror it in `qiling/cli.py` if user-facing - ([cli.md](cli.md)); keep the composition order intact. -- **Adding a hook type**: add the `Protocol`, the `hook_*` method, and a - `_hook_*_cb` dispatcher in `qiling/core_hooks.py`, preserving - `QL_HOOK_BLOCK` handling. -- **Adding an arch/OS/loader**: only the naming derivation in - `qiling/utils.py` factories and the enums in `qiling/const.py` change here; - the implementation belongs to the owning module. -- **Do not** import from `qiling/os`, `qiling/loader`, `qiling/arch` at - module level in `qiling/core.py` beyond `TYPE_CHECKING`; the factories exist to - keep that direction one-way. -- Improvement candidates (proposals, not accepted work): replace the - `type()` checks in hook dispatch with `isinstance` and document return - semantics (`qiling/core_hooks.py:186`, `:209`); replace the hard-coded - guard-page floor `0x9000000` (`qiling/core.py:525`) with a profile value. - Success check: existing suites stay green. - -## Open Gaps / Roadmap - -- No dedicated unit test for the hook engine or for `save/restore` outside - MCU; coverage is indirect. -- The thumb workaround in `emu_start` (`qiling/core.py:759`) depends on - Unicorn behavior; revisit only as part of a Unicorn upgrade (project-wide - event, see the root deviations). -- `ChangeLog` stops at 1.4.6 (`ChangeLog:4`) while `pyproject.toml:4` is - 1.4.12.dev0. -- Feature wishlist lives in GitHub issue - [#333](https://github.com/qilingframework/qiling/issues/333); the hybrid - kernel roadmap is in `TODO.md` (owned by [kernel-proxy.md](kernel-proxy.md)). diff --git a/ARCHITECTURE/debugger.md b/ARCHITECTURE/debugger.md deleted file mode 100644 index 7335ff95a..000000000 --- a/ARCHITECTURE/debugger.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -eatmycode_version: "1.2.0" ---- - -# Debugger — GDB server and Qdb - -## Goal - -Let users debug emulated targets: a GDB remote-serial-protocol server so -any GDB/IDA/lldb front end can attach cross-architecture, and Qdb, a -built-in interactive debugger with stepping, branch prediction, and -record/replay reverse debugging. Owns both front ends and their per-arch -tables; it must not own register or memory semantics. No roadmap -milestone applies; maturity-based status. - -## Status - -`done` — Qdb covered by `tests/test_qdb.py` (observed: `Ran 6 tests … OK`, -including RISC-V 32/64); the GDB server by `tests/test_debugger.py` -(observed: `Ran 7 tests … OK`, ~71 s, drives scripted clients on port 9999). - -## Code Structure - -| File | Role | -| ---- | ---- | -| `qiling/debugger/debugger.py` | Base `QlDebugger` | -| `qiling/debugger/gdb/gdb.py` | `QlGdb`: RSP packet handlers (`handle_c/g/G/m/M/q/v/s/X/Z/z`), `GdbSerialConn` socket transport | -| `qiling/debugger/gdb/utils.py` | `QlGdbUtils`: breakpoint table and the per-instruction `dbg_hook` servicing breakpoints, stepping, and async interrupts | -| `qiling/debugger/gdb/xmlregs.py`, `gdb/xml//` | `QlGdbFeatures`: target-description XML per arch (a8086, x86, x8664, arm, arm64, cortex_m, mips, ppc, riscv, riscv64) | -| `qiling/debugger/qdb/qdb.py` | `QlQdb(Cmd, QlDebugger)`: command loop (`do_run/step_in/step_over/continue/backward/breakpoint/…`) | -| `qiling/debugger/qdb/arch/` | Per-arch register naming/aliases (arm, cortex-m, intel, mips, riscv) | -| `qiling/debugger/qdb/branch_predictor/` | Predicts branch targets for step/next per arch | -| `qiling/debugger/qdb/render/` | Register/stack/disasm context rendering per arch | -| `qiling/debugger/qdb/utils.py`, `helper.py`, `context.py`, `misc.py`, `const.py` | Factories (`setup_branch_predictor`, `setup_context_render`), `SnapshotManager` for rr, expression helper | - -## Language and Conventions - -Python; root rules apply. Local patterns: - -- GDB packet handlers are closures named `handle_` inside - `QlGdb.run` (`qiling/debugger/gdb/gdb.py:248-762`) returning a reply - string; stop replies use `SIGTRAP` (`:49`) for breakpoints and steps and - `SIGINT` for async interrupts (`:262-265`). -- Qdb commands are `do_` methods on `QlQdb` - (`qiling/debugger/qdb/qdb.py:204-691`). -- Per-arch Qdb support is three parallel class families selected by - `QL_ARCH` maps in `qiling/debugger/qdb/utils.py:109-146`; RISC-V was - added following that pattern (`qiling/debugger/qdb/arch/arch_riscv.py:11`, - `branch_predictor_riscv.py`, `render_riscv.py`). -- `qiling/debugger/qdb/branch_predictor/__init__.py:13-17` contains - tab-indented lines; do not add more. - -## Design and Invariants - -- **Activation**: `ql.debugger = True | "gdb" | "gdb:HOST:PORT" | "qdb" | - "qdb:rr" | "qdb: