Feature request: alter and insert on a let-local, in place (growing a local copies every step today) #950

Summary

Rayfall cannot update a let local or a lambda parameter in place.

• alter rejects a local target (domain: alter: local bindings are not supported). That rejection came with the alter on a let-local inside a lambda writes a global (local unchanged, same-named global clobbered) #810 fix (fix(alter): reject local targets before global writes #846). Before it, the write silently went to a same-named global.
• The functional forms copy. (let P (insert P rows)) copies the whole table on every call, so a loop that grows a local table is quadratic.

The only linear way to grow or amend a value is therefore a global. #821 made that fast for dotted globals. It also means any engine that accumulates state, such as a backtester's run state, has to keep it in globals, while the other q-family runtimes keep it in locals.

Reproduction

The loop runs 500 instruments, with a 500-row append and one 10-element amend per step, on rayforce-git 2.10.0.r217.g014b98ad:

(set N 500)
(set syms (take [a b c d e f g h i j] N))
(set mkP (fn [] (table [ts sym qty] (list (as 'I64 []) (as 'SYM []) (as 'I64 [])))))

;; global: linear
(set runG (fn [steps]
  (set bt {}) (set bt.P (mkP)) (set bt.q (take 0 N)) (set k 0)
  (while (< k steps)
    (insert 'bt.P (list (take k N) syms bt.q))
    (alter 'bt.q set (til 10) (+ (take k 10) 1))
    (set k (+ k 1)))
  (count bt.P)))

;; local: no in-place form, so the table is copied every step
(set runL (fn [steps]
  (let P (mkP)) (let q (take 0 N)) (let k 0)
  (while (< k steps)
    (let P (insert P (list (take k N) syms q)))
    (let q (+ q (concat (take 1 10) (take 0 (- N 10)))))
    (let k (+ k 1)))
  (count P)))

Other q-family runtimes

The same loop in each runtime's idiom (P,:rows and q[i]:v on a local), timed at 1,000 → 2,000 steps, in ms:

peachq runs on the Rayforce runtime, so the runtime already supports this. Its modassign_eval (src/qlang/eval/q_eval.c) takes the value out of the local's slot (q_env_local_take), so the binding is its only owner. It then appends in place (q_join_amend) and stores the result back.

Proposal

Let alter and insert amend a let local or a parameter of the enclosing lambda in place, with the same forms they take for globals:

(let P (mkP))
(insert 'P rows)              ; appends to the local P
(alter 'q set idx vals)       ; amends the local q
(alter 'o concat vals)

• Resolution as for reads: the innermost local binding of the name, then a global. Writing the local keeps alter on a let-local inside a lambda writes a global (local unchanged, same-named global clobbered) #810's bug fixed: a local target never reaches a global.
• When the local is the value's only owner, the update happens in place. When the value is shared (a closure, or another binding holding it), copy-on-write keeps the other holder's value unchanged, as ray_cow already does.
• An unbound name stays an error.

Use case

The Rayfall backtester keeps a run's whole state (positions, fills, records) in a global dictionary, because only globals can be updated in place. Local state would let one process hold several runs, and it would make the engine reentrant. It is also what a real pmap (#948) would need: in kdb+ and amber, parallel workers may read globals but not assign them (noupdate), so per-run state must live in locals.

A dictionary passed into a function and returned is quadratic in every runtime above (kdb+ 120 → 1,258 ms), so the request is only about locals, not about functional state passing.