Problematic load/save semantics for args in the server

[jeremyfowers]

The Lemonade API spec allows for models to be load``ed with highly customized CLI options, which we call *_args (e.g., `llamacpp_args`). I will refer to these as "args" for short. We also support saving args, as well as server-provided default args.

In trying to implement GUI3, I have discovered that it is very difficult for a Lemonade client to navigate the semantics of loading and saving with args. Here are the problems in lemond on main as of this writing:

Architectural Defaults

There is a JSON file that defines 6 per-model-architecture default sampling parameters (temp, top-k, etc.). These are automatically applied to models, and there is no way to disable this.

Users can override these args, for example by passing their own --temp. However, omitting one such value semantically means "I want the architectural default". So there is no way to override one arg and pass no value for the others.

Saved Value Merge on Load

Lemonade allows for args to be saved, for example --no-mmap --threads 8. These saved args are merged with args passed in the load body (ie, the load body does not wholesale replace the saved args).

So if I saved --no-mmap --threads 8 and then pass --threads 4 in the load body, the result is --no-mmap threads 4. There is no way for me to pass my --threads 4 and tell Lemonade to omit the --no-mmap except by clearing the entire saved entry for that model.

Saved Value Merge on Save

Saving new args does not wholesale replace previously saved args. For example, if I saved --no-mmap and then save --threads 8, now my saved args are --no-mmap --threads 8.

Hidden and Conditional Defaults

The WrapedServer C++ for Lemonade also automatically applies a variety of args at runtime, some of which are static (like -ngl 99) and some of which are dynamic (like --no-mmap on APUs only). These arguments are not disclosed by the API, not even the /models/{id}/options endpoint, except that there are some cases where lemond will raise an HTTP error if the load body contradicts one of these args.

Conclusion

It is a bug that clients are unable to exercise control, or gain visibility, over the args they are loading. We should propose a comprehensive solution to allow full control and visibility.

[jeremyfowers]

@bitgamma @fl0rianr want your thoughts here, did I miss anything or get anything wrong?

[jeremyfowers]

My proposal:

1. If a load request body passes _args it should wholesale replace any saved args (no merge)
2. If a save request body passes _args those should wholesale replace any saved args (no merge)
3. There should be a way to opt-out of architectural and WrapedServer-specific defaults, such that the _args in a load body are the only _args that get applied whatsoever
4. WrapedServer-specific _args should appear in /model/{id}/options effective settings results.

[bitgamma]

My proposal:

1. If a load request body passes _args it should wholesale replace any saved args (no merge)
2. If a save request body passes _args those should wholesale replace any saved args (no merge)
3. There should be a way to opt-out of architectural and WrapedServer-specific defaults, such that the _args in a load body are the only _args that get applied whatsoever
4. WrapedServer-specific _args should appear in /model/{id}/options effective settings results.

So your proposal is to remove argument merging as soon as the user defines arguments in recipe_options/load parameter? That was the situation until a few months ago and everybody was complaining about having to repeat common arguments in every model. The "merge_args" flag was the compromise to keep both factions happy, because it effectively disables ALL merging (and if it doesn't, it is a bug)

PS: Sorry I accidentally clicked "edit" instead of "quote reply" the first time

[bitgamma]

Since I think llamacpp_args can be very roughly split into "per-model stuff" and "per-machine stuff", I think that removing the feature that allows you to change global, machine related settings is not a good compromise and I think many people rely on that feature. For example I have "-b 2048 -ub 1024 -np 1" in my backend settings and do not want to have to repeat it in every single model (because there isn't a single model that I run with exactly the default settings) and most importantly if I add/remove something there, I want all models to immediately reflect the changes.

So my counter-proposal is to do what Jeremy said BUT the backend args always get merged (unless merge_args is false) with the recipe_options args.

Why I think this is a good compromise:

1. backend args are empty on a new installation. The GUI can hide the entire field and "merge" flag if these are empty, effectively mirroring what @jeremyfowers proposed
2. advanced users still have the ability to define per-server (per-backend actually) defaults that can be updated at-once while being able to set per-model settings and get to see the final line (backend args are already queriable)

Additionally, we could consider if the architectural defaults are worth it or if they turned out to be non that great and we could try to remove any non-vital "overridable" argument from WrappedServer. For llamacpp we currently have 4.

1. --reasoning-format auto: matches the llamacpp default, can be removed with no actual consequence
2. --no-ui: disables llamacpp webui, not really necessary
3. --load-mode none: when on iGPU, (was --no-mmap before). This won't really be necessary with newer builds at the new default (auto) already matches this behavior
4. --spec-type dflash/mtp. This is the only useful one, we should probably keep this.

[fl0rianr]

I think the disagreement here mostly disappears if we separate machine/backend args from model args.

Jeremy is right that a client which explicitly sends llamacpp_args needs to be able to replace the model-specific state. Otherwise the client can never really know what it is loading.

At the same time, I think bitgamma is right that backend args like

-b 2048 -ub 1024 -np 1

are machine-level settings and should not have to be repeated for every model.

So I would picture the current problem roughly like this:

backend / machine args
        │
        ▼
architecture defaults
        │
        ▼
saved model args
        │
        ▼
load request args
        │
        ▼
merge everything
        │
        ▼
WrappedServer adds more defaults
        │
        ▼
actual backend command line

The client has two problems here:

1. sending args does not mean "these are my model args", it means "merge these into whatever was already there"

2. even after resolving the options, WrappedServer can still add more args that are not visible through the API

I think a cleaner model would be:

BACKEND / MACHINE
global user defaults
-b 2048 -ub 1024 -np 1
        │
        ▼
MODEL
architecture + saved per-model args
        │
        ▼
REQUEST
explicit args for this load
        │
        ▼
RUNTIME DEFAULTS
automatic Lemonade decisions
        │
        ▼
effective args exposed by API
        │
        ▼
backend starts with exactly those args

With one important rule:

no request *_args
    -> inherit model args normally
request *_args present

-> replace the model-level *_args

but keep backend/machine args
merge_args = false

-> do not inherit/merge any lower-level custom args

Example:

backend:

-b 2048 -ub 1024 -np 1

saved model args:

--no-mmap --threads 8
load request:

--threads 4

With normal merging enabled, I would expect:

-b 2048 -ub 1024 -np 1 --threads 4

not:

-b 2048 -ub 1024 -np 1 --no-mmap --threads 4

The request replaced the model args, but did not remove the machine defaults.

And if the client sends:

{

"merge_args": false,

"llamacpp_args": "--threads 4"

}

then I would expect the custom args to be exactly:

--threads 4

apart from arguments Lemonade fundamentally has to manage itself (-m, --port, etc.).

That also makes merge_args=false a useful escape hatch for clients that want full control.

---

For saving, I think we should keep persistence separate from load-time merging.

If the saved value is currently:

--no-mmap

and I save:

--threads 8

I would expect the stored value to become:

--threads 8

not:

--no-mmap --threads 8

llamacpp_args is one stored field. Merging different sources can happen when resolving a load, but saving a new value should not secretly merge it with the previous value.

---

The last piece is the hidden WrappedServer defaults.

I think the important invariant should be:

GET /models/{id}/options

│

▼

effective

│

▼

POST /load with effective

│

▼

same effective backend configuration

In other words, no overridable argument should appear after the "effective options" have been computed.

Things like the iGPU --load-mode none behavior or --spec-type can still be automatic, but they should be part of the resolved/effective settings and ideally carry their source/reason, e.g.

--load-mode none

source: runtime default

reason: iGPU detected

That would also make GUI3 much easier because it could distinguish:

Backend default

Architecture default

Saved for this model

Explicit for this load

Automatically selected by Lemonade

instead of only seeing one final merged string.

So my proposed semantics would basically be:

Request	Custom args result
no _args, merge=true	backend + model/default args
_args=X, merge=true	backend + X
no _args, merge=false	no inherited custom args
_args=X, merge=false	X only
save _args=X	stored value becomes X

Does this separation between backend/machine scope and model/request scope make sense to you both? A bigger change than I thought at first, but I think it preserves the global-settings use case while giving API clients deterministic control.