perf(pool): size the default pool from the CPUs the process may run on

docs/docs/architecture/memory.md

@@ -89,7 +89,7 @@ Pools are normally backed by anonymous `mmap` — fast, lazy-committed by the ke
 The fallback is transparent to every `ray_alloc` caller. Pages never written stay as holes in the sparse backing file, so the over-allocate-and-trim alignment trick costs zero real disk space; only the pages you actually touch consume disk pages. On heap teardown the fd is closed and the tempfile is unlinked, so the swap directory doesn't accumulate orphans.
 
 - **Configurable swap directory.** The directory is resolved as `RAY_HEAP_SWAP` → `TMPDIR` → `/tmp`. `/tmp` is the default rather than the working directory, which may be read-only or shared. Trailing slash is added automatically. The chosen directory must exist and be writable by the running process.
-- **Tempfile naming.** Format is `rayheap_<pid>_<heap_id>_<counter>.dat`. Files are opened `O_EXCL` so no clashes between concurrent processes; the counter is per-process atomic so no clashes within a process.
+- **Tempfile naming.** Pools use `rayheap_<pid>_<heap_id>_<counter>.dat`, direct blocks `raydirect_<pid>_<heap_id>_<counter>.dat`. The pid keeps processes of one PID namespace apart, and the counter is a per-process atomic. Files are opened `O_EXCL`, so a name that already exists fails that allocation rather than sharing the file.
 - **POSIX only today.** The fallback path uses `open` / `ftruncate` / `mmap MAP_SHARED` / `munmap` / `unlink`. Windows pools currently take only the anonymous `VirtualAlloc` path.
 - **Distinct from block offloading.** Block offloading (see [Block Offloading](offloading.md)) streams pre-existing parted-table data through queries one segment at a time. The file-backed pool fallback handles fresh anonymous allocations that exceed RAM. Both let workloads exceed RAM, but at different layers and for different shapes of work.
 

docs/docs/architecture/pipeline.md

@@ -207,6 +207,8 @@ Rayforce uses a global thread pool (`ray_pool_t`) for parallel execution. Querie
 
 Each dispatch splits its tasks into one contiguous range per worker (the main thread included); a worker runs its range in order and, once it is empty, steals half of another worker's remaining range from the tail, so an idle thread never waits on a busy one and the rows a worker touches are sequential. Tasks of one dispatch may therefore run in any order and on any worker: operators place results by task index, and any early stop (a `take:` cutoff, a top-N bound) is derived from row positions or values, never from "the earlier tasks have run". The scheduler is selected once at pool creation (`RAY_POOL_STEAL`, default `1`; `0` restores the shared claim cursor for A/B comparison) and `RAY_POOL_TRACE=1` prints the own-versus-stolen task histogram per worker to stderr at pool destroy; see [REPL Reference](../language/repl.md#environment-variables).
 
+The default pool has one thread per CPU of the process affinity mask (Linux; the online CPU count elsewhere), or per CPU of a cgroup CPU quota, rounded up, when that is smaller. Threads are not tied to CPUs: the scheduler places them within the mask.
+
 ### Radix-Partitioned Hash Join
 
 Hash joins use adaptive radix partitioning to ensure each partition's hash table fits in L2 cache. The number of radix bits adapts between `RAY_JOIN_MIN_RADIX` (2, producing 4 partitions) and `RAY_JOIN_MAX_RADIX` (14, producing 16K partitions) based on input size, targeting a per-partition working set of `RAY_JOIN_L2_TARGET` (256 KB).

docs/docs/guides/instances.md

@@ -0,0 +1,72 @@
+# Running Several Instances
+
+Several rayforce processes can run on one machine. Each process sizes its thread pool and its memory for what it can see, and it does not know about the others. Give every instance its own share of CPUs and memory, its own port and journal, and keep each database written by one process at a time.
+
+## CPUs
+
+Choose the CPUs of each instance when you start it: `taskset -c LIST`, `docker run --cpuset-cpus=LIST`, or `CPUAffinity=` in a systemd unit. Every thread of the instance, and every program it starts with `.sys.exec`, stays within that list.
+
+On Linux an instance sizes its thread pool to its list: one thread per CPU of it, the main thread included. The operating system keeps the threads within the list and moves them between its CPUs as it sees fit; rayforce does not tie a thread to one CPU. On other platforms the pool has one thread per online CPU.
+
+```bash
+taskset -c 0,2,4,6,24,26,28,30 rayforce -p 5001 &
+taskset -c 8,10,12,14,32,34,36,38 rayforce -p 5002 &
+```
+
+**Choosing the lists.** Read the machine's layout from `lscpu -e` (columns `NODE`, `SOCKET`, `CORE`), or from `/sys/devices/system/cpu/cpuN/topology/thread_siblings_list` and `/sys/devices/system/node/nodeN/cpulist`.
+
+- Give each instance whole cores: both hardware threads of a core go to the same instance. Two instances on the two threads of one core slow each other down.
+- Keep an instance on one NUMA node when its CPU count fits in one.
+- CPU numbering does not follow the layout on every machine. On a two-socket machine that numbers sockets alternately (even CPUs on socket 0, odd on socket 1) with CPU `k` and `k+24` on one core, the ranges `0-7`, `8-15`, … `40-47` put every instance on both sockets, and the last three ranges on the hyperthreads of the first three. For six instances of 8 threads on that machine use `0,2,4,6,24,26,28,30`, `8,10,12,14,32,34,36,38`, `16,18,20,22,40,42,44,46`, `1,3,5,7,25,27,29,31`, `9,11,13,15,33,35,37,39` and `17,19,21,23,41,43,45,47`.
+
+**CPU quota.** `docker run --cpus=N` and systemd `CPUQuota=` limit CPU time, not CPUs: the instance may still run on every CPU. Its default pool then has one thread per CPU of the quota, rounded up. Give it `--cpuset-cpus` or `CPUAffinity=` as well to choose its CPUs.
+
+**Do not:**
+
+- start two instances on overlapping lists: they share those CPUs;
+- start instances without a list and leave each at its default size: every instance takes every CPU;
+- pass `-c` larger than the list: the threads take turns on its CPUs (rayforce prints a warning).
+
+`(.sys.info)` reports the CPUs of the instance (`cores`) and its pool size (`workers`); `taskset -p PID` shows its list.
+
+## Memory
+
+Each instance treats the machine's whole physical RAM, or the memory limit of its container or cgroup when one is set, as its own. It moves its working set to disk only when its own memory would pass that amount, so several busy instances without limits of their own can together exhaust RAM before any of them spills. Start every instance with `-m SIZE` and keep the sizes, plus room for the page cache, within RAM:
+
+```bash
+taskset -c 0-11 rayforce -m 40G -p 5001 &
+taskset -c 12-23 rayforce -m 40G -p 5002 &
+```
+
+`-m` also bounds the freed memory an idle instance keeps for reuse (a sixteenth of SIZE).
+
+Instances that read the same tables share the operating system's page cache, so the sum of their resident sizes overstates the memory in use.
+
+## Ports, journals and spill files
+
+- **Port:** each instance listens on its own `-p` port. A second instance on a busy port exits with an error.
+- **Journal:** each instance needs its own journal base (`-l /path/instance1`). Two processes appending to one journal break its framing, and recovery stops at the first broken entry. Never run `.log.roll`, `.log.snapshot` or `.log.purge` on a journal another process writes.
+- **Spill files:** names carry the process id and are removed as soon as they are mapped, so instances never share one. Point `RAY_HEAP_SWAP` at a disk with room for every instance that may spill at once.
+
+## Sharing a database
+
+**Readers.** Any number of instances can read the same splayed or partitioned tables. A reader sees either the whole previous version of a table or the whole new one, and a table it has already loaded stays valid when another process rewrites it.
+
+**One writer per database.** Only one process at a time may write a database: `.db.splayed.set`, a `.csv` or `.parquet` import into an existing root, or a live `insert` into a partitioned table followed by its rollover. All tables and partitions of a database share the root's symbol file.
+
+- When two processes write the same database, a write that races another one fails with a `corrupt` error. Nothing on disk is damaged. If the failing process holds tables it loaded from that database, its later reads and writes of the database fail until it drops them.
+- Two processes that update one table by reading it, changing it and writing it back lose each other's updates; the last write wins.
+- A partitioned reader fails while a new partition is being written, because the partition directory exists before its schema does. Write a new partition under a dot directory of the root, passing the root's symbol file, and rename it into place; dot directories are skipped when the partitions are listed:
+
+```lisp
+(.db.splayed.set "/db/.stage/2026.10.02/t/" t "/db/.sym")
+(.sys.exec "mv /db/.stage/2026.10.02 /db/2026.10.02")
+```
+
+**Files.** Never change table files in place: no `cp` over them, no `rsync --inplace`, no `truncate`, no editors. Instances that have the table open see the change at once, its stored statistics no longer match the data, and a shortened file crashes them with `SIGBUS`. Replace tables with `.db.splayed.set`, or write a new file and rename it over the old one. The same applies to files written with `write`, `write-bytes` and `hnsw-save`, which rewrite their target in place.
+
+**Versions.** All instances that share a database must run the same rayforce version; upgrade them together.
+
+## Interactive sessions
+
+The REPL rewrites `~/.rayforce_history` when it exits, so with several interactive sessions only the history of the last one to exit is kept.

docs/docs/guides/storage.md

@@ -69,7 +69,7 @@ ray_err_t ray_sym_save(const char* path);
 ray_err_t ray_sym_load(const char* path);
 ```
 
-The symbol file uses an append-only format: new symbols are appended on save, and loading merges them into the running table. File locking (`flock` on POSIX, `LockFileEx` on Windows) ensures safe concurrent access.
+`ray_sym_save` writes the whole runtime table to a temporary file and renames it over the target, so a reader never sees a partial file; there is no merge with another writer's file, so one process writes a given file. Loading merges the file into the running table.
 
 In typical usage, you do not call these directly — the splayed and partitioned table functions handle symbol persistence automatically.
 

docs/docs/language/repl.md

@@ -17,7 +17,9 @@ The `rayforce` binary operates in three modes depending on how it is invoked:
 echo '(+ 1 2)' | ./rayforce
 ```
 
-Interactive mode is detected automatically when stdin is a terminal. On startup the REPL prints a banner with the version, CPU, memory, logical CPU count, and worker count, then shows the `‣` prompt. By default the pool includes every online logical CPU, including SMT threads. Use `-c N` to choose a total worker count explicitly: N must contain decimal digits only; `0` selects the automatic size, and `1..1023` sets the total count, including the main thread. Values outside this range are rejected before the pool starts. Individual operations may use fewer tasks when their workload or memory budget requires it. Use `-m SIZE` (`--mem`; a byte count or a `K`/`M`/`G`/`T` suffix) to set the RAM budget: anonymous memory past it spills to disk instead of risking the OOM killer. `(.sys.mem)` reports the effective value as [`anon-watermark`](../namespaces/sys.md#sys-mem).
+Interactive mode is detected automatically when stdin is a terminal. On startup the REPL prints a banner with the version, CPU, memory, logical CPU count, and worker count, then shows the `‣` prompt. By default the pool includes every logical CPU the process may run on (all online CPUs, unless `taskset` or a cpuset narrowed them), including SMT threads. Use `-c N` to choose a total worker count explicitly: N must contain decimal digits only; `0` selects the automatic size, and `1..1023` sets the total count, including the main thread. Values outside this range are rejected before the pool starts. Individual operations may use fewer tasks when their workload or memory budget requires it. Use `-m SIZE` (`--mem`; a byte count or a `K`/`M`/`G`/`T` suffix) to set the RAM budget: anonymous memory past it spills to disk instead of risking the OOM killer. `(.sys.mem)` reports the effective value as [`anon-watermark`](../namespaces/sys.md#sys-mem).
+
+Under a cgroup CPU quota (`docker --cpus`, systemd `CPUQuota=`) the default pool has one thread per CPU of the quota, rounded up. A `-c` larger than the allowed CPUs prints a warning. See [Running Several Instances](../guides/instances.md) for choosing CPUs per instance.
 
 ### Environment Variables
 

docs/docs/namespaces/sys.md

@@ -11,7 +11,7 @@ Process-level introspection (build, memory, host info) and command-style operati
 |---|---|---|---|
 | [`.sys.args`](#sys-args) | variadic | — | Command-line arguments as a typed dict. |
 | [`.sys.build`](#sys-build) | nullary | — | Version + build date as a dict. |
-| [`.sys.info`](#sys-info) | nullary | — | Host and process facts: cores, page size, total memory, pid, hostname. |
+| [`.sys.info`](#sys-info) | nullary | — | Host and process facts: cores, page size, total memory, pool size, pid, hostname. |
 | [`.sys.mem`](#sys-mem) | nullary | — | Allocator statistics. |
 | [`.sys.prof`](#sys-prof) | nullary | — | Last profiled query's per-step statistics as a table. |
 | [`.sys.querylog`](#sys-querylog) | variadic | — | Ambient per-query statistics ring as a table. |
@@ -75,12 +75,15 @@ Signature: `(.sys.build)`. Returns a dict with `version` (string) and `build-dat
 
 ## `.sys.info` { #sys-info }
 
-Signature: `(.sys.info)`. Returns `{cores: i64, page-size: i64, total-mem: i64, pid: i64, hostname: str}` on every platform (on Windows from `GetSystemInfo` / `GlobalMemoryStatusEx`).
+Signature: `(.sys.info)`. Returns `{cores: i64, page-size: i64, total-mem: i64, workers: i64, pid: i64, hostname: str}` on every platform (on Windows from `GetSystemInfo` / `GlobalMemoryStatusEx`).
+
+- `cores`: the CPUs this process may run on. On Linux that is its affinity mask, which `taskset`, a cpuset or `docker --cpuset-cpus` may narrow below the machine's CPU count.
+- `workers`: the threads of the execution pool, main included; before the pool exists, the size it will get. `.sys.info` does not create the pool (see [Running Several Instances](../guides/instances.md)).
 
 ```lisp
 (.sys.info)
 ;; => {cores: 10, page-size: 16384, total-mem: 68719476736,
-;;     pid: 48213, hostname: "tp-prod-01"}
+;;     workers: 10, pid: 48213, hostname: "tp-prod-01"}
 ```
 
 `pid` and `hostname` describe *this process*, not the machine, and exist so an

docs/docs/storage/index.md

@@ -222,7 +222,9 @@ The global symbol intern table maps strings to integer IDs. When saving data to
 
 ### Append-Only .sym Files
 
-Symbol files use an append-only format. New symbols are appended to the end of the file without rewriting existing entries. This makes concurrent writes safe and enables incremental updates.
+A table's symbol file, which `.db.splayed.set` and the importers write, is append-only: new symbols are appended to the end of the file without rewriting existing entries, so a symbol's position never changes and updates are incremental. One process writes a database at a time: a writer whose view of the file is out of date fails with `RAY_ERR_CORRUPT` instead of overwriting another writer's symbols (see [Running Several Instances](../guides/instances.md)).
+
+The C functions below work on the runtime's global symbol table instead. `ray_sym_save` writes the whole table to a temporary file and renames it over the target, and `ray_sym_load` merges a file into the running table.
 
 | Function | Description |
 |---|---|
@@ -243,7 +245,7 @@ ray_sym_load("db/.sym");
 ### Concurrency and Integrity
 
 - **File locking** — `flock()` on POSIX, `LockFileEx()` on Windows. Multiple processes can safely read the symbol file; writes acquire an exclusive lock.
-- **Corruption detection** — symbol files include checksums. If a file is truncated or corrupted, `ray_sym_load()` returns `RAY_ERR_CORRUPT` and leaves the in-memory table unchanged.
+- **Corruption detection** — symbol files carry no checksum. The loader checks the magic, the entry count and every entry's length; a truncated or malformed file makes `ray_sym_load()` return `RAY_ERR_CORRUPT`.
 - **Arena backing** — interned strings are allocated from a dedicated arena (`ray_arena_t`), making bulk allocation fast and ensuring all strings are freed together when the symbol table is destroyed.
 
 ## CSV Import and Export
@@ -412,7 +414,7 @@ For every table present in **any** partition, ensures **every** partition has it
 
 Symbol tables are persisted automatically when you use `.db.splayed.set`. A `.sym` file is written into the table directory containing all interned symbol strings. When loading with `.db.splayed.get` or `.db.parted.get`, the symbol table is loaded first so that symbol columns decode correctly.
 
-At the C API level, `ray_sym_save(path)` and `ray_sym_load(path)` handle persistence directly. The format is append-only — new symbols are appended without rewriting existing entries, making concurrent access safe.
+At the C API level, `ray_sym_save(path)` and `ray_sym_load(path)` handle persistence directly. `ray_sym_save` writes the whole runtime table to a temporary file and renames it over `path`; one process writes a given file.
 
 ## Working Example: Round-Trip
 

mkdocs.yml

@@ -167,6 +167,7 @@ nav:
           - IPC Guide: docs/guides/ipc.md
           - Error Handling: docs/guides/errors.md
           - Memory & Monitoring: docs/guides/memory.md
+          - Running Several Instances: docs/guides/instances.md
           - Type Casting: docs/guides/types.md
           - Timespan: docs/guides/timespan.md
       - Architecture:

src/app/main.c

@@ -313,6 +313,13 @@ int main(int argc, char** argv) {
      * (file load, REPL eval, builtins).  If -c wasn't given, leave the
      * pool to its lazy default on first use. */
     if (n_cores >= 0) {
+        /* the CPUs this process may run on, or its CPU quota when smaller */
+        uint32_t ncpu = ray_pool_default_total();
+        if ((uint32_t)n_cores > ncpu)
+            fprintf(stderr, "warning: -c %d gives %d threads, more than the %u CPU%s this process "
+                            "may use, and they take turns; use -c %u, or drop -c to size the pool "
+                            "to the CPUs\n",
+                    n_cores, n_cores, ncpu, ncpu == 1 ? "" : "s", ncpu);
         ray_err_t err = ray_pool_init_total((uint32_t)n_cores);
         if (err != RAY_OK) {
             fprintf(stderr, "error: ray_pool_init_total(%d) failed (%d)\n",

src/app/repl.c

@@ -396,11 +396,7 @@ static void print_banner(void) {
     char cpu[256];
     get_cpu_name(cpu, sizeof(cpu));
     int64_t mem_mb = get_total_mem_mb();
-#if defined(RAY_OS_WINDOWS)
-    int ncores = (int)ray_thread_count();
-#else
-    int ncores = (int)sysconf(_SC_NPROCESSORS_ONLN);
-#endif
+    int ncores = (int)ray_cpu_set_count();   /* the CPUs this process may run on */
 
     /* "Using" count reflects the actual worker-pool size, not ncores.
      * ray_pool_get() is a lazy initializer — callers might not have