[AdrianSosic]

Fixes #793 and related workarounds emerging from it.

❗ The following is a human-written (i.e. not AI-generated) summary ❗

Summary of the key changes and design rationales

Problem

Choice of dataframe backend

• We use pandas and therefore also force users to use pandas (Dataframe backend lock-in #795)
• Internal pandas logic is error prone (ambiguous syntax, weird index behavior, etc)
• Everything is computed eagerly (Eager search space materialization #796), which causes several problems:
  • Architectural hacks (e.g. workarounds for config validation without materialization)
  • Infinite spaces (e.g. with unbound sequence parameter) generally not possible
  • No lazy filtering possible, which makes handling large spaces inflexible (No mechanism to navigate large/infinite spaces #798)
  • General pandas performance problems for large dataframes

Solution

We use narwhals as a dataframe abstraction layer:

• Inspired by polars' powerful expression syntax (also add more user-facing options for filtering, e.g. custom constraints, including serializability)
• Users can chose their favorite dataframe backend
• We get polars performance for free when polars is used as backend, or we can explicitly require it for certain performance critical paths
• Allows us to easily add mechanics for handling large/infinite spaces using lazy execution
• Potentially no more OOM errors due to streaming engine (default since polars 2.0)
• We always get a functional "vanilla" implementation that works for any existing and future search space type for free, e.g. from_simplex would automatically work even without specialized implementation
• Additional bonus: narwhals is entirely dependency-free and has a "stable" API! This means:
  • Much slimmer dependency footprint for baybe
  • No more hotfix PRs that work around issues from indirect dependencies, and also ideally no more changes due to narhwals itself

---

Problem

The attribute structure of the search space classes is completely messed up:

• Unenforced invariants (Unenforced search space invariants #794)
• Unnecessary attributes and constructor arguments (e.g. empty_encoding or exposed comp_rep)
• Continuous constraints spread across several attributes, requiring the user to manually classify them
• Discrete constraints are only meaningful for the from_product constructor and have no meaning as a general attribute. Similarly, the concept of "constrained" search space is not well defined and neither are the is_constraint properties.
• Comp rep is computed eagerly, causing several problems:
  • Computational (e.g. problematic for validation-only path, some recommenders don't require it)
  • Encodings must be available eagerly and assumed static (e.g. fixed columns and order, and they cannot be adjusted/learned during the process), which blocks the user from providing parameter values that are outside the pre-defined value range
  • Surrogates see a larger construct than what is actually needed, with potential downsides on the statistical properties and computation speed. For example, in a constrained recommendation context when only 2 out of 10 hypothetical substances are present, columns from the 10-based comp rep will be used even though the 2-based version could be much more compact (e.g. many correlated columns for the 2 available substances).

Solution

Complete redesign of the attribute structure. Most importantly:

• Minimal (i.e. non-redudant) represenation where invariants are enforce statically by design
• Class-based Candidate approach addresses the design problem at it's core: different space definitions require different attribute structures (e.g. constraints only relevant for product spaces)
• Encoding happens lazily, right before the transition to the surrogate, encoding only "what is there" at that moment
• ❗ Still missing: Comp-rep dependent steps (e.g. computation of scaling ranges, dropping of columns by correlation) needs to happen lazily as well, i.e. we need to get rid of assuming a static comp rep. This will ultimately enable:
  • Users no longer need to pre-specify the full list of parameter values, i.e. they can add measurment data for to-be-encoded parameters even if the parameter list do not contain these values as long as the encoder knows how to handle them (makes active_values sort of obsolete but still useful from a user perspective)
  • Better surrogate performance (both in terms of speed and in the statistical sense, since focusing only on the relevant part of the space)
• ❗ Still missing: All discrete parameter should have an "encoder" attribute. This will:
  • Simplify the class structure a lot, i.e. parameters effectively only differ by their encoder
  • For numeric paramaters, the decoder defaults to an identity encoder but, when provided, can be used to desribe 1-to-N mappings (e.g. degree of freedom is angle but computational representation uses x/y)

---

Problem

Campaign attribute structure is messed up:

• Unmaintained (i.e. logically broken) fields like n_fits_done
• Meta-data mixed with experimental data (e.g. FitNr)
• No good representation of batches other than through semi-documented meta data columns (e.g. BatchNr)
• Coupling to eager-materialized search space exp rep for tracking metadata (e.g. RECOMMENDED) and resulting weird workarounds like _FilteredSubspaceDiscrete

Solution

Redesign of the campaign attribute structure. Most importantly:

• Separate meta-data from experimental data
• Introduce a clear representation for batches in form of separated internally stored dataframes
• Metadata is now tracked cleanly in form of dedicated internal dataframes. Allow-flag-dependent search space filtering happes lazily on the fly.
• ❗ Still missing: A clean and lazy SubspaceDiscrete.filter method, that allows us to easily and cheaply (!) adjust discrete spaces with a single call using a flexible Selector approach (i.e. using all input types that qualify as a row-selector, such as discrete constraints, dataframe expressions, dataframes that specify included rows via (anti)-joins, or anything that can be converted to any of the previous)

---

Problem

The call chain down the recommenders is messed up:

• There are legacy arguments that are either no longer needed or just used as a workaround required by the _FilteredDiscreteSubspace approach.
• The call stack jumps between Bayesian and Base recommenders via various super calls and disabled validation, making the overall logic super hard to follow, especially for newcomers

Solution

• With the lazy search space approach, _FilteredDiscreteSubspace becomes obsolete (since search spaces can now be natively and cheaply filtered whenever necessary), and thus the workarounds are no longer needed. The call signatures can thus be redesigned from the ground up.
• ❗ Still missing: A final cleanup that streamlines the call chain, avoiding opaque super calls

---

Other things missing

• Some dataframe-based helpers and functionality that is less relevant to the default execution path is only narwhalified at the API boundary but still uses pandas internally --> needs to be rewritten in narwhals but can be done at any time without major implications
• After constraint refactoring is complete: add missing polars/narwhals logic and refactor interface, i.e. get rid of get_invalid approaches and instead use an expression-based approach. Proposal: a to_expr() -> nw.Expr method for internal use in expression based contexts and a main __call__(df: IntoFrameT): --> IntoFrameT entry point for directly applying the constraint to a given (lazy) dataframe
• Dropping pandas entirely as a main dependency (can remain for tests tho)

---

State of the PR

PRs merged

• Refactor Campaign #821
• Add CandidatesProtocol #800
• Refactor searchspace attributes #778
• Fix Serialization #833
• Recommendation Dataframes #832
• Deprecate SubspaceDiscrete.constraints #835
• Refactor recommendation mechanics #838
• Refactor candidates interface #840
• Wire CandidatesProtocol into SubspaceDiscrete #842
• Backend Settings #865
• Narwhalify DiscreteParameter.transform #862
• Narwhalify transformation execution chain #867
• Narwhalify recommendation layer #887

Upcoming PRs

• Narwhalify campaign #917
• Filter mechanism for discrete candidate contexts #944
• Narwhalify model layer #913
• Add sequence parameter #866
• Lazify get_candidates #915
• Narwhalify conditions #963
• Narwhalify constraints #953

---

To be decided

• How to handle conflicting backends, e.g. when a campaign already stores a pandas-dataframe but then a users adds a polars batch? 1) Ignore and let narwhals error surface (this is an unlikely scenario) 2) Introduce a ._backend property and validate against such cases 3) make Campaign generic and only complain on a typing level
• I vote that the is_constrained property of the search space classes should be dropped (and have already done so)! Reasons:
  • SubspaceContinuous.is_constrained used to contain non-trivial logic but with the new design it's equivalent to a simple if self.constraints call
  • SubspaceDiscrete.is_constrained was very much meaningless since the constraints attribute itself was flawed (Deprecate SubspaceDiscrete.constraints #835). And with both the legacy and new design, it's not obvious what it actually means! Can provide more insights if needed.
  • Having an is_constrained attribute only on some of the search space classes is asymmetric
• SubspaceDiscrete.batch_constraints vs DiscreteBatchConstraint name conflict. Options:
  • rename batch_constraints to something else, e.g. recommendation_constraints. However, batch_constraints is already a generic term. Perhaps we should rather ...
  • rename DiscreteBatchConstraint to something more specific. The latter is actually rather a name for an abstract base class if we decide to add more batch-level constraints, and does not convey anything about what it does. Options would be in the direction of DiscreteSharedValueConstraint
• Let's fully kick the semi-document fields with very imprecise/questionable semantics:n_batches_done and n_fits_done (already done)
• Let's fully kick mixed-in meta data columns FitNr and BatchNr (already done)
• Tricky: Scaling approach: shall we use complete unfiltered space, filtered space or policy-generated subset? Decision depends heavily on what is cheap and possible in a large and dynamically spanned space but also what makes conceptually sense (i.e. should adding a hypothetical candidate that is removed through a filter, e.g. by policy or active values, change the induced scaling?). Also impacts methods like comp_rep_bounds, which now behave differently compared to main (Refactor candidates interface #840 (comment)). At the same time, the discrete version of comp_rep_bounds may be dropped entirely since effectively unused at the moment (it's called in optimize_acqf_mixed but the discrete bounds should actually not matter since the candidates are explicitly piped in as separate argument)
  My opinion: as proposed in PR summary above, I believe that any "comp rep" processing should always just only look at "what is there" at that moment. Specifically: the effective candidate range for the recommendation + the measurement range. Can elaborate more if necessary.
• Decorrelation: same problem as for scaling above
• Namings such as:
  • EmptyCandidates vs NullCandidates
  • TableCandidates vs ...
  • encode vs transform vs computational representation terminology. Transform is currently used for continuous-valued objects (e.g. target transforms) but also for in the parameter/searchspace context. However, the latter also uses encoding (e.g. discrete parameters have specified encodings, we have a correspondingly named base class, ...). We somehow need to make this more precise. Perhaps: transform for conti mappings and encode for discrete mappings? Also covers things like the _encoding_table methods etc.
• ...
• The entire "batch constraint" situation:
  • Batch constraints are actually not related to the search space (i.e. they impose no filtering condition) but are rather a requirement for the recommender and should probably be moved there. This is reflected by the fact how they are handled in the new code version, i.e. they currently live as isolated attributes in SubspaceDiscrete and do not enter any of its mechanisms.
  • BatchConstraint should probably rather be the name for a potential base class specifying batch constraints, while the current one should have a clearer name that really specifies what it imposes of the batch (such as SyncedValuesConstraint or similar – there were some good ideas flowing around somewhere in the issue where I first pointed this out to Alex, would need to look it up)
• Should we make any guarantees on (preserving) pandas indices? Mention in changelog/docs? If preserved --> tests? Right now, it's done for transformations (including tests).
• Replacing constraints with filters in ProductCandidates:
  • agreed?
  • what about constructors, such as from_product
  • do we want to allow "eager filters" in the first place (e.g. Boolean masks)?

TODOs

• Freeze (sub)space classes once all other (blocking) issues are resolved
• Rework or remove _resolve_default_base in baybe/surrogates/gaussian_process/core.py, can ideally just be a factory call, see Add Per-Parameter Convenience Kernel Override #904 (comment)
• Remember to update the surrogate fodler structure, connected to cahgnes/additions from the TFPR branch Add TFPR recommender #918, see Add TFPR recommender #918 (comment)