diff --git a/Cargo.toml b/Cargo.toml index dbad253..fc52a8f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["rayforce-sys", "rayforce"] [workspace.package] -version = "1.3.0" +version = "1.3.1" edition = "2021" license = "MIT" authors = ["RayforceDB"] @@ -16,4 +16,4 @@ categories = ["database", "api-bindings"] rust-version = "1.74" [workspace.dependencies] -rayforce-sys = { path = "rayforce-sys", version = "1.3.0" } +rayforce-sys = { path = "rayforce-sys", version = "1.3.1" } diff --git a/README.md b/README.md index 4453962..f6feee8 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ To move the pin, move both: ```sh git -C rayforce-sys/vendor/rayforce fetch --tags -git -C rayforce-sys/vendor/rayforce checkout v2.11.0 +git -C rayforce-sys/vendor/rayforce checkout v2.11.5 git add rayforce-sys/vendor/rayforce git -C rayforce-sys/vendor/rayforce rev-parse --short=7 HEAD # CORE_COMMIT diff --git a/docs/docs/content/CHANGELOG.md b/docs/docs/content/CHANGELOG.md index cb993a7..e2dcda3 100644 --- a/docs/docs/content/CHANGELOG.md +++ b/docs/docs/content/CHANGELOG.md @@ -3,6 +3,85 @@ All notable changes to `rayforce` are documented here. This project adheres to [Semantic Versioning](https://semver.org). +## 1.3.1 + +### Changed + +- **The vendored core is v2.11.5 (`5d5f47c`) and `rayforce-q` is 2.4.0 + (`df8ced9`)**, from v2.11.0 and 2.3.0. `include/rayforce.h` is unchanged, and + the `bindings.rs` bindgen generates changes only in the opaque + `ray_runtime_s`, 48 to 560 bytes for the limiting cgroup's directory; at that + size bindgen no longer derives `Copy` / `Clone` for it, and the crate only + ever holds it by pointer. The nineteen `INTERNAL_FNS` signatures are + unchanged. Of the private headers bindgen reads, `src/lang/eval.h`, + `src/lang/env.h`, `src/core/runtime.h` and `src/ops/ops.h` grow for compiled + closures and for the row provenance of helpers called from a query, and + `src/lang/internal.h` gains `ray_null_class`; none is in `INTERNAL_FNS`. The + new sources `src/ops/select_merge.c`, `src/ops/symcmp.c`, + `src/table/symgrp.c` and `src/table/symimp.c` are picked up by the Makefile's + `src/*/*.c` wildcard and the packaged globs, and link against nothing new; + the Makefile changes only its fuzz targets. rayforce-q has pushed no 2.4.0 + tag, so it is pinned by its "Release 2.4.0" commit. `q.c` is unchanged; 2.4.0 + adds `(.q.handle)`, the connection whose pushed frame a handler is running + for, which `q_server.c` now records around each dispatch. The builtin is + registered by rayforce-q's `embed/rayforce_q.c`, which this crate does not + compile, and `q_conn_current` is not declared in `rayforce-sys`, so it does + not reach this crate yet. + +- **`rayforce-sys` erases both spellings of `_Atomic` for bindgen's parse.** + v2.11.5's `src/lang/env.h` declares `extern _Atomic uint64_t g_global_epoch`, + an atomic spelled as a qualifier at file scope, which bindgen 0.70 cannot + type and aborts on ("Couldn't resolve constant type"), and which the + function-like `-D_Atomic(T)=T` define could not reach. `_Atomic` is now + defined as `__volatile` and `__volatile` as a function-like macro, so + `_Atomic(T)` reads as `T` and `_Atomic T` as a volatile `T`. `rayforce-sys` + 1.3.0 cannot generate its bindings against a v2.11.5 core given through + `RAYFORCE_SRC`. Only bindgen's parse is affected: no allowlisted item holds + an atomic, and the core is compiled by its own Makefile. + +- **The v2.11.5 engine deltas that reach this crate outside `eval`.** + `Table::insert_row` appends an empty string to a string column, which raised + `vec_append: str vectors use ray_str_vec_append`. `Table::upsert_columns` on + a table with mapped columns, such as one from `Table::load_splayed`, checks + each value column's length against the key's and fails with `length`, where + a shorter column was read past its end. `Table::save_splayed` keeps one copy + of a replaced table on disk instead of about three: only the selected + generation is kept, and the first publish removes the root column files of + the layout it retires. `Table::load_splayed` whose generation is removed + while it opens retries for as long as the table keeps changing (at least + eight times, for up to 60 seconds) and stops on an interrupt, where it + failed with an I/O error after eight attempts; `Table::load_parted` reuses + unchanged partitions across mounts. `Value::deserialize` refuses a lambda + whose closure is not a dict of symbol keys to a list of values with `type`. + Every block the core allocates outside its pools, the string arenas and hash + tables of the symbol table and column domains included, now counts toward + the spill watermark, and the default watermark sits a quarter below the RAM + the process may use (at least 3 GiB below, at most half of it) rather than + at all of it. Spill files are opened close-on-exec, so a `.sys.exec` child + no longer holds them. + +- **What changes through `eval` and the query builders.** Closures compile to + bytecode, their captures bound as locals, and `return` ends a lambda on its + first, interpreted call as it does once compiled. A lambda called from a + query reads the query's source columns, and a `by:` key the planner cannot + lower is evaluated once over the table and grouped on as a computed key. + `in` compares a string element of a collection as one value (it answered + per character), works over string columns inside a query (it raised `nyi`) + and answers an empty boolean vector for an empty left operand, so + `(where (in keys held))` no longer raises when `keys` is empty. Nulls of + different types are different values in `in`, `find`, `distinct` and join + keys, while numeric nulls still meet across widths, as `1` meets `1i`; + `not-in` excludes null symbols. A dict row given to `upsert` may leave out + non-key columns, and a row that leaves out a list column gets the null + object, where reading the cell aborted. Ordered symbol columns compare by + their text; a sort or take over a grouping with no outputs works on a + computed key; a multi-key `select` takes a distinct count beside an + expression aggregate; nested selects are composed before planning. Parquet + imports go column by column when memory is short, hash indexes store 32-bit + row ids and slots below 2^32 rows, scans read ahead on cold storage, and + `sys-mapped` counts file mappings in whole pages, where it drifted below + zero. + ## 1.3.0 ### Added diff --git a/docs/docs/content/get-started/installation.md b/docs/docs/content/get-started/installation.md index 528b409..5e145ca 100644 --- a/docs/docs/content/get-started/installation.md +++ b/docs/docs/content/get-started/installation.md @@ -77,11 +77,11 @@ Moving the pin means moving the submodule and the constants together: ```sh # 1. Move the submodule to the new tag. git -C rayforce-sys/vendor/rayforce fetch --tags -git -C rayforce-sys/vendor/rayforce checkout v2.11.0 +git -C rayforce-sys/vendor/rayforce checkout v2.11.5 git add rayforce-sys/vendor/rayforce # 2. Read back the values build.rs must stamp. -git -C rayforce-sys/vendor/rayforce describe --tags --exact-match # -> v2.11.0 +git -C rayforce-sys/vendor/rayforce describe --tags --exact-match # -> v2.11.5 git -C rayforce-sys/vendor/rayforce rev-parse --short=7 HEAD # -> e.g. 1a2b3c4 ``` @@ -89,7 +89,7 @@ Then edit `rayforce-sys/build.rs` to match — `CORE_VERSION` is the tag without its leading `v`: ```rust -const CORE_VERSION: &str = "2.11.0"; +const CORE_VERSION: &str = "2.11.5"; const CORE_COMMIT: &str = "1a2b3c4"; ``` diff --git a/rayforce-sys/build.rs b/rayforce-sys/build.rs index a2fd3e7..087555d 100644 --- a/rayforce-sys/build.rs +++ b/rayforce-sys/build.rs @@ -24,7 +24,7 @@ use std::process::Command; /// /// Must match the tag `vendor/rayforce` is pinned to. CI asserts the two agree; /// see the "Check vendored core pin" step in `.github/workflows/ci.yml`. -const CORE_VERSION: &str = "2.11.0"; +const CORE_VERSION: &str = "2.11.5"; /// Commit the `vendor/rayforce` submodule is pinned to, stamped alongside /// [`CORE_VERSION`]. Also checked by CI's "Check vendored core pin" step. @@ -35,7 +35,7 @@ const CORE_VERSION: &str = "2.11.0"; /// under OUT_DIR, an unset value does not fall back to "unknown" — it silently /// reports the HEAD of whatever unrelated repository happens to enclose the /// build directory. -const CORE_COMMIT: &str = "dc9b12a"; +const CORE_COMMIT: &str = "5d5f47c"; /// Warning flags for the vendored core build — the core's own `WARNS` /// (`Makefile:35`) minus `-Werror`. Consumers compile this with whatever @@ -195,32 +195,40 @@ fn main() { .clang_arg(format!("-I{}", core.join("src").display())) // bindgen 0.70 cannot resolve the type of a C11 `_Atomic` variable and // aborts the whole parse with "Couldn't resolve constant type". The - // parse reaches the core's atomics only through `lang/internal.h`, the - // only header declaring ray_{set,get}_splayed_fn / ray_get_parted_fn, - // which includes `mem/heap.h`, which includes `core/platform.h`. - // Defining the keyword away costs nothing. The atomics found there are - // the file scope globals `ray_heap_pending_merge` (`mem/heap.h:553`) - // and `ray_parallel_flag` (`core/platform.h:195`) and the `foreign` - // field of `ray_heap_t` (`mem/heap.h:533`). Neither global is - // allowlisted, `ray_heap_t` is reachable from nothing that is, and - // `include/rayforce.h` never says `_Atomic` — so no generated type - // contains one, and no layout bindgen emits can shift. This only - // affects bindgen's parse; the core is compiled by its own Makefile. + // core spells atomics two ways, the specifier `_Atomic(T)` and the + // qualifier `_Atomic T`, and no single macro can match both: a + // function-like `_Atomic(T)` is not expanded where no parenthesis + // follows. So `_Atomic` is defined as `__volatile`, and `__volatile` + // as a function-like macro. A specifier rescans to `__volatile(T)` and + // then to plain `T`; a qualifier, with no parenthesis to invoke it, + // stays `__volatile T`, an ordinary volatile that bindgen parses. No + // header in the parse writes `__volatile(`; inline asm, the one place + // a volatile keyword meets a parenthesis, conventionally spells it + // `__volatile__` or `volatile`, and neither is touched. // - // `core/ipc.h` brings a fourth through `core/poll.h`: the `code` field - // of `struct ray_poll` (`core/poll.h:116`), spelled as the qualifier - // `_Atomic int64_t`, which this function-like define does not touch. - // bindgen parses that form without aborting but types the field `u64`; - // it stays out of the bindings only because `ray_poll` is opaque below. - .clang_arg("-D_Atomic(T)=T") + // The specifiers come in through `lang/internal.h`, the only header + // declaring ray_{set,get}_splayed_fn / ray_get_parted_fn, which + // includes `mem/heap.h`, which includes `core/platform.h`: the file + // scope globals `ray_heap_pending_merge` (`mem/heap.h:609`) and + // `ray_parallel_flag` (`core/platform.h:231`) and the `foreign` field + // of `ray_heap_t` (`mem/heap.h:587`). The qualifiers are the `code` + // field of `struct ray_poll` (`core/poll.h:116`), which `core/ipc.h` + // brings in, and the global `g_global_epoch` (`lang/env.h:172`). No + // global is allowlisted, `ray_heap_t` is reachable from nothing that + // is, `ray_poll` is opaque below, and `include/rayforce.h` never says + // `_Atomic` — so no generated type contains one, and no layout bindgen + // emits can shift. This only affects bindgen's parse; the core is + // compiled by its own Makefile. + .clang_arg("-D_Atomic=__volatile") + .clang_arg("-D__volatile(T)=T") // Everything the public header declares. This bound is load-bearing: - // the private headers added below declare ~660 functions and ~100 RAY_* + // the private headers added below declare ~700 functions and ~100 RAY_* // constants between them, so a blanket `ray_.*` would drag in the whole // internal surface. Anchored loosely because the staged path lives under // OUT_DIR, which may itself contain regex metacharacters. .allowlist_file(".*/include/rayforce\\.h") // The public header leaves ray_runtime_s incomplete (`rayforce.h:771`) - // and `core/runtime.h:117` completes it. Left alone, bindgen would + // and `core/runtime.h:156` completes it. Left alone, bindgen would // publish the runtime internals — ray_vm_t and friends, ~67 KB of // private layout that would then churn on every core bump. Opaque // keeps it a handle, which is all the public API ever passes around. diff --git a/rayforce-sys/vendor/rayforce b/rayforce-sys/vendor/rayforce index dc9b12a..5d5f47c 160000 --- a/rayforce-sys/vendor/rayforce +++ b/rayforce-sys/vendor/rayforce @@ -1 +1 @@ -Subproject commit dc9b12a61046cb12435710b57ff61ea681658f19 +Subproject commit 5d5f47c294b87e8757abb8633592ecdbda4659e9 diff --git a/rayforce-sys/vendor/rayforce-q b/rayforce-sys/vendor/rayforce-q index f4378a4..df8ced9 160000 --- a/rayforce-sys/vendor/rayforce-q +++ b/rayforce-sys/vendor/rayforce-q @@ -1 +1 @@ -Subproject commit f4378a4fc544aab46886d62c6cc3844d77d7641a +Subproject commit df8ced99a21d4bd2545f7ec18cce1798c52a3f17