Feat/core 2.11.5

Cargo.toml

@@ -3,7 +3,7 @@ resolver = "2"
 members = ["rayforce-sys", "rayforce"]
 
 [workspace.package]
-version = "1.3.0"
+version = "1.3.1"
 edition = "2021"
 license = "MIT"
 authors = ["RayforceDB"]
@@ -16,4 +16,4 @@ categories = ["database", "api-bindings"]
 rust-version = "1.74"
 
 [workspace.dependencies]
-rayforce-sys = { path = "rayforce-sys", version = "1.3.0" }
+rayforce-sys = { path = "rayforce-sys", version = "1.3.1" }

README.md

@@ -127,7 +127,7 @@ To move the pin, move both:
 
 ```sh
 git -C rayforce-sys/vendor/rayforce fetch --tags
-git -C rayforce-sys/vendor/rayforce checkout v2.11.0
+git -C rayforce-sys/vendor/rayforce checkout v2.11.5
 git add rayforce-sys/vendor/rayforce
 
 git -C rayforce-sys/vendor/rayforce rev-parse --short=7 HEAD   # CORE_COMMIT

docs/docs/content/CHANGELOG.md

@@ -3,6 +3,85 @@
 All notable changes to `rayforce` are documented here. This project adheres to
 [Semantic Versioning](https://semver.org).
 
+## 1.3.1
+
+### Changed
+
+- **The vendored core is v2.11.5 (`5d5f47c`) and `rayforce-q` is 2.4.0
+  (`df8ced9`)**, from v2.11.0 and 2.3.0. `include/rayforce.h` is unchanged, and
+  the `bindings.rs` bindgen generates changes only in the opaque
+  `ray_runtime_s`, 48 to 560 bytes for the limiting cgroup's directory; at that
+  size bindgen no longer derives `Copy` / `Clone` for it, and the crate only
+  ever holds it by pointer. The nineteen `INTERNAL_FNS` signatures are
+  unchanged. Of the private headers bindgen reads, `src/lang/eval.h`,
+  `src/lang/env.h`, `src/core/runtime.h` and `src/ops/ops.h` grow for compiled
+  closures and for the row provenance of helpers called from a query, and
+  `src/lang/internal.h` gains `ray_null_class`; none is in `INTERNAL_FNS`. The
+  new sources `src/ops/select_merge.c`, `src/ops/symcmp.c`,
+  `src/table/symgrp.c` and `src/table/symimp.c` are picked up by the Makefile's
+  `src/*/*.c` wildcard and the packaged globs, and link against nothing new;
+  the Makefile changes only its fuzz targets. rayforce-q has pushed no 2.4.0
+  tag, so it is pinned by its "Release 2.4.0" commit. `q.c` is unchanged; 2.4.0
+  adds `(.q.handle)`, the connection whose pushed frame a handler is running
+  for, which `q_server.c` now records around each dispatch. The builtin is
+  registered by rayforce-q's `embed/rayforce_q.c`, which this crate does not
+  compile, and `q_conn_current` is not declared in `rayforce-sys`, so it does
+  not reach this crate yet.
+
+- **`rayforce-sys` erases both spellings of `_Atomic` for bindgen's parse.**
+  v2.11.5's `src/lang/env.h` declares `extern _Atomic uint64_t g_global_epoch`,
+  an atomic spelled as a qualifier at file scope, which bindgen 0.70 cannot
+  type and aborts on ("Couldn't resolve constant type"), and which the
+  function-like `-D_Atomic(T)=T` define could not reach. `_Atomic` is now
+  defined as `__volatile` and `__volatile` as a function-like macro, so
+  `_Atomic(T)` reads as `T` and `_Atomic T` as a volatile `T`. `rayforce-sys`
+  1.3.0 cannot generate its bindings against a v2.11.5 core given through
+  `RAYFORCE_SRC`. Only bindgen's parse is affected: no allowlisted item holds
+  an atomic, and the core is compiled by its own Makefile.
+
+- **The v2.11.5 engine deltas that reach this crate outside `eval`.**
+  `Table::insert_row` appends an empty string to a string column, which raised
+  `vec_append: str vectors use ray_str_vec_append`. `Table::upsert_columns` on
+  a table with mapped columns, such as one from `Table::load_splayed`, checks
+  each value column's length against the key's and fails with `length`, where
+  a shorter column was read past its end. `Table::save_splayed` keeps one copy
+  of a replaced table on disk instead of about three: only the selected
+  generation is kept, and the first publish removes the root column files of
+  the layout it retires. `Table::load_splayed` whose generation is removed
+  while it opens retries for as long as the table keeps changing (at least
+  eight times, for up to 60 seconds) and stops on an interrupt, where it
+  failed with an I/O error after eight attempts; `Table::load_parted` reuses
+  unchanged partitions across mounts. `Value::deserialize` refuses a lambda
+  whose closure is not a dict of symbol keys to a list of values with `type`.
+  Every block the core allocates outside its pools, the string arenas and hash
+  tables of the symbol table and column domains included, now counts toward
+  the spill watermark, and the default watermark sits a quarter below the RAM
+  the process may use (at least 3 GiB below, at most half of it) rather than
+  at all of it. Spill files are opened close-on-exec, so a `.sys.exec` child
+  no longer holds them.
+
+- **What changes through `eval` and the query builders.** Closures compile to
+  bytecode, their captures bound as locals, and `return` ends a lambda on its
+  first, interpreted call as it does once compiled. A lambda called from a
+  query reads the query's source columns, and a `by:` key the planner cannot
+  lower is evaluated once over the table and grouped on as a computed key.
+  `in` compares a string element of a collection as one value (it answered
+  per character), works over string columns inside a query (it raised `nyi`)
+  and answers an empty boolean vector for an empty left operand, so
+  `(where (in keys held))` no longer raises when `keys` is empty. Nulls of
+  different types are different values in `in`, `find`, `distinct` and join
+  keys, while numeric nulls still meet across widths, as `1` meets `1i`;
+  `not-in` excludes null symbols. A dict row given to `upsert` may leave out
+  non-key columns, and a row that leaves out a list column gets the null
+  object, where reading the cell aborted. Ordered symbol columns compare by
+  their text; a sort or take over a grouping with no outputs works on a
+  computed key; a multi-key `select` takes a distinct count beside an
+  expression aggregate; nested selects are composed before planning. Parquet
+  imports go column by column when memory is short, hash indexes store 32-bit
+  row ids and slots below 2^32 rows, scans read ahead on cold storage, and
+  `sys-mapped` counts file mappings in whole pages, where it drifted below
+  zero.
+
 ## 1.3.0
 
 ### Added

docs/docs/content/get-started/installation.md

@@ -77,19 +77,19 @@ Moving the pin means moving the submodule and the constants together:
 ```sh
 # 1. Move the submodule to the new tag.
 git -C rayforce-sys/vendor/rayforce fetch --tags
-git -C rayforce-sys/vendor/rayforce checkout v2.11.0
+git -C rayforce-sys/vendor/rayforce checkout v2.11.5
 git add rayforce-sys/vendor/rayforce
 
 # 2. Read back the values build.rs must stamp.
-git -C rayforce-sys/vendor/rayforce describe --tags --exact-match   # -> v2.11.0
+git -C rayforce-sys/vendor/rayforce describe --tags --exact-match   # -> v2.11.5
 git -C rayforce-sys/vendor/rayforce rev-parse --short=7 HEAD        # -> e.g. 1a2b3c4
 ```
 
 Then edit `rayforce-sys/build.rs` to match — `CORE_VERSION` is the tag without
 its leading `v`:
 
 ```rust
-const CORE_VERSION: &str = "2.11.0";
+const CORE_VERSION: &str = "2.11.5";
 const CORE_COMMIT: &str = "1a2b3c4";
 ```
 

rayforce-sys/build.rs

@@ -24,7 +24,7 @@ use std::process::Command;
 ///
 /// Must match the tag `vendor/rayforce` is pinned to. CI asserts the two agree;
 /// see the "Check vendored core pin" step in `.github/workflows/ci.yml`.
-const CORE_VERSION: &str = "2.11.0";
+const CORE_VERSION: &str = "2.11.5";
 
 /// Commit the `vendor/rayforce` submodule is pinned to, stamped alongside
 /// [`CORE_VERSION`]. Also checked by CI's "Check vendored core pin" step.
@@ -35,7 +35,7 @@ const CORE_VERSION: &str = "2.11.0";
 /// under OUT_DIR, an unset value does not fall back to "unknown" — it silently
 /// reports the HEAD of whatever unrelated repository happens to enclose the
 /// build directory.
-const CORE_COMMIT: &str = "dc9b12a";
+const CORE_COMMIT: &str = "5d5f47c";
 
 /// Warning flags for the vendored core build — the core's own `WARNS`
 /// (`Makefile:35`) minus `-Werror`. Consumers compile this with whatever
@@ -195,32 +195,40 @@ fn main() {
         .clang_arg(format!("-I{}", core.join("src").display()))
         // bindgen 0.70 cannot resolve the type of a C11 `_Atomic` variable and
         // aborts the whole parse with "Couldn't resolve constant type". The
-        // parse reaches the core's atomics only through `lang/internal.h`, the
-        // only header declaring ray_{set,get}_splayed_fn / ray_get_parted_fn,
-        // which includes `mem/heap.h`, which includes `core/platform.h`.
-        // Defining the keyword away costs nothing. The atomics found there are
-        // the file scope globals `ray_heap_pending_merge` (`mem/heap.h:553`)
-        // and `ray_parallel_flag` (`core/platform.h:195`) and the `foreign`
-        // field of `ray_heap_t` (`mem/heap.h:533`). Neither global is
-        // allowlisted, `ray_heap_t` is reachable from nothing that is, and
-        // `include/rayforce.h` never says `_Atomic` — so no generated type
-        // contains one, and no layout bindgen emits can shift. This only
-        // affects bindgen's parse; the core is compiled by its own Makefile.
+        // core spells atomics two ways, the specifier `_Atomic(T)` and the
+        // qualifier `_Atomic T`, and no single macro can match both: a
+        // function-like `_Atomic(T)` is not expanded where no parenthesis
+        // follows. So `_Atomic` is defined as `__volatile`, and `__volatile`
+        // as a function-like macro. A specifier rescans to `__volatile(T)` and
+        // then to plain `T`; a qualifier, with no parenthesis to invoke it,
+        // stays `__volatile T`, an ordinary volatile that bindgen parses. No
+        // header in the parse writes `__volatile(`; inline asm, the one place
+        // a volatile keyword meets a parenthesis, conventionally spells it
+        // `__volatile__` or `volatile`, and neither is touched.
         //
-        // `core/ipc.h` brings a fourth through `core/poll.h`: the `code` field
-        // of `struct ray_poll` (`core/poll.h:116`), spelled as the qualifier
-        // `_Atomic int64_t`, which this function-like define does not touch.
-        // bindgen parses that form without aborting but types the field `u64`;
-        // it stays out of the bindings only because `ray_poll` is opaque below.
-        .clang_arg("-D_Atomic(T)=T")
+        // The specifiers come in through `lang/internal.h`, the only header
+        // declaring ray_{set,get}_splayed_fn / ray_get_parted_fn, which
+        // includes `mem/heap.h`, which includes `core/platform.h`: the file
+        // scope globals `ray_heap_pending_merge` (`mem/heap.h:609`) and
+        // `ray_parallel_flag` (`core/platform.h:231`) and the `foreign` field
+        // of `ray_heap_t` (`mem/heap.h:587`). The qualifiers are the `code`
+        // field of `struct ray_poll` (`core/poll.h:116`), which `core/ipc.h`
+        // brings in, and the global `g_global_epoch` (`lang/env.h:172`). No
+        // global is allowlisted, `ray_heap_t` is reachable from nothing that
+        // is, `ray_poll` is opaque below, and `include/rayforce.h` never says
+        // `_Atomic` — so no generated type contains one, and no layout bindgen
+        // emits can shift. This only affects bindgen's parse; the core is
+        // compiled by its own Makefile.
+        .clang_arg("-D_Atomic=__volatile")
+        .clang_arg("-D__volatile(T)=T")
         // Everything the public header declares. This bound is load-bearing:
-        // the private headers added below declare ~660 functions and ~100 RAY_*
+        // the private headers added below declare ~700 functions and ~100 RAY_*
         // constants between them, so a blanket `ray_.*` would drag in the whole
         // internal surface. Anchored loosely because the staged path lives under
         // OUT_DIR, which may itself contain regex metacharacters.
         .allowlist_file(".*/include/rayforce\\.h")
         // The public header leaves ray_runtime_s incomplete (`rayforce.h:771`)
-        // and `core/runtime.h:117` completes it. Left alone, bindgen would
+        // and `core/runtime.h:156` completes it. Left alone, bindgen would
         // publish the runtime internals — ray_vm_t and friends, ~67 KB of
         // private layout that would then churn on every core bump. Opaque
         // keeps it a handle, which is all the public API ever passes around.