Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions asap-planner-rs/src/bin/candidate_gen_dump.rs
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ fn main() -> anyhow::Result<()> {
query_string: q.clone(),
t_repeat_ms: qg.repetition_delay_ms,
accuracy_sla: qg.controller_options.accuracy_sla,
latency_sla: qg.controller_options.latency_sla,
latency_sla_ms: qg.controller_options.latency_sla_ms,
})
})
.collect();
Expand All @@ -71,13 +71,13 @@ fn main() -> anyhow::Result<()> {

for (i, aqe) in aqes.iter().enumerate() {
println!(
"\n--- Item #{i}: metric={} stat={:?} range={}ms T={}ms accuracy_sla={} latency_sla={} freq={:.4}Hz ---",
"\n--- Item #{i}: metric={} stat={:?} range={}ms T={}ms accuracy_sla={} latency_sla_ms={:?} freq={:.4}Hz ---",
aqe.requirements.metric,
aqe.requirements.statistics,
aqe.requirements.data_range_ms,
aqe.t_repeat_ms,
aqe.accuracy_sla,
aqe.latency_sla,
aqe.latency_sla_ms,
aqe.query_frequency_hz,
);
println!(" queries: {:?}", aqe.query_strings);
Expand Down
149 changes: 140 additions & 9 deletions asap-planner-rs/src/config/input.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,16 +32,16 @@ pub struct ControllerConfig {
}

impl ControllerConfig {
/// Warn if any query group has both SLAs at 0.0 (the serde Default),
/// which indicates `controller_options` was omitted from the config.
/// Warn if any query group still has the serde-default SLAs, which
/// indicates `controller_options` was omitted from the config.
pub fn warn_default_slas(&self) {
for qg in &self.query_groups {
let opts = &qg.controller_options;
if opts.accuracy_sla == 0.0 && opts.latency_sla == 0.0 {
if opts.accuracy_sla == 0.0 && opts.latency_sla_ms.is_none() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warn_default_slas treats accuracy_sla == 0.0 && latency_sla_ms.is_none() as "controller_options missing". Omitting latency_sla_ms is now normal, so a valid explicit {accuracy_sla: 0.0} triggers a false warning. Making QueryGroup.controller_options an Option<ControllerOptions> would make this exact instead of a guess.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The false positive needs an explicit accuracy_sla: 0.0, which means "accept any error" and is almost never intended, so a warning there is arguably still useful. Making controller_options an Option would touch every caller for a log line. Happy to revisit if the warning becomes noisy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With omitted latency_sla_ms now a legitimate "no limit", accuracy_sla == 0.0 && latency_sla_ms.is_none() false-positives on an explicit controller_options: {accuracy_sla: 0.0}. Making controller_options: Option<ControllerOptions> and warning on None would detect omission directly instead of via sentinel values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier thread on this line: leaving it. A false positive needs an explicit accuracy_sla: 0.0 ("accept any error"), and Option<ControllerOptions> would touch every caller for a log line.

warn!(
query_group_id = ?qg.id,
"controller_options not set in query group; \
accuracy_sla=0.0 and latency_sla=0.0 will be used — \
accuracy_sla=0.0 and no latency_sla_ms limit will be used — \
add controller_options to your config"
);
}
Expand All @@ -63,6 +63,7 @@ impl ControllerConfig {
}

#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deny_unknown_fields is added to QueryGroup but not to SQLQueryGroup, ElasticDSLQueryGroup, or SQLControllerConfig. A typo'd group key (e.g. misspelled repetition_delay_ms, stray group-level latency_sla) is still silently dropped for SQL/Elastic configs while the same mistake is rejected for PromQL.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8d79801. SQLControllerConfig, SQLQueryGroup, ElasticDSLControllerConfig and ElasticDSLQueryGroup now deny unknown fields (sql_and_elastic_configs_reject_unknown_group_keys). Existing fixtures all still parse.

pub struct QueryGroup {
pub id: Option<u32>,
pub queries: Vec<String>,
Expand All @@ -79,11 +80,13 @@ pub struct QueryGroup {
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ControllerOptions now has deny_unknown_fields, but QueryGroup doesn't. A typo like controler_options: still parses, the whole SLA block falls back to its default, and the latency ceiling is silently lost. Consider adding deny_unknown_fields here too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0d3f0a8. QueryGroup now has deny_unknown_fields (test rejects_misspelled_controller_options). Experiment configs keep client_options inside each query group, so the asap-tools controller-input writer now strips just that key per group. Any other unknown key still reaches the planner and fails.

#[derive(Debug, Clone, Deserialize, Default)]
#[serde(deny_unknown_fields)]
pub struct ControllerOptions {
#[serde(deserialize_with = "deserialize_finite_f64")]
pub accuracy_sla: f64,
#[serde(deserialize_with = "deserialize_finite_f64")]
pub latency_sla: f64,
/// Maximum modeled query latency in milliseconds; `None` means no limit.
#[serde(default, deserialize_with = "deserialize_optional_positive_f64")]
pub latency_sla_ms: Option<f64>,
}

fn deserialize_finite_f64<'de, D>(deserializer: D) -> Result<f64, D::Error>
Expand All @@ -98,6 +101,18 @@ where
}
}

fn deserialize_optional_positive_f64<'de, D>(deserializer: D) -> Result<Option<f64>, D::Error>
where
D: Deserializer<'de>,
{
match Option::<f64>::deserialize(deserializer)? {
Some(value) if !(value.is_finite() && value > 0.0) => Err(serde::de::Error::custom(
"must be a finite number greater than zero",
)),
value => Ok(value),
}
}

fn deserialize_positive_u64<'de, D>(deserializer: D) -> Result<u64, D::Error>
where
D: Deserializer<'de>,
Expand Down Expand Up @@ -211,6 +226,7 @@ pub struct HllParams {
}

#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SQLControllerConfig {
pub query_groups: Vec<SQLQueryGroup>,
pub tables: Vec<TableDefinition>,
Expand All @@ -220,6 +236,7 @@ pub struct SQLControllerConfig {
}

#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SQLQueryGroup {
pub id: Option<u32>,
pub queries: Vec<String>,
Expand All @@ -237,13 +254,15 @@ pub struct TableDefinition {
}

#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ElasticDSLControllerConfig {
pub query_groups: Vec<ElasticDSLQueryGroup>,
pub sketch_parameters: Option<SketchParameterOverrides>,
pub aggregate_cleanup: Option<AggregateCleanupConfig>,
}

#[derive(Debug, Clone, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ElasticDSLQueryGroup {
pub id: Option<u32>,
pub queries: Vec<String>,
Expand All @@ -265,7 +284,6 @@ query_groups:
repetition_delay_ms: 60000
controller_options:
accuracy_sla: .nan
latency_sla: .inf
"#;

let error = serde_yaml::from_str::<ControllerConfig>(yaml)
Expand All @@ -283,13 +301,126 @@ query_groups:
repetition_delay_ms: 60000
controller_options:
accuracy_sla: 0.99
latency_sla: 1.0
latency_sla_ms: 250.0
"#;

let config: ControllerConfig = serde_yaml::from_str(yaml).unwrap();
let options = &config.query_groups[0].controller_options;
assert_eq!(options.accuracy_sla, 0.99);
assert_eq!(options.latency_sla, 1.0);
assert_eq!(options.latency_sla_ms, Some(250.0));
}

#[test]
fn omitted_latency_sla_ms_means_no_limit() {
let yaml = r#"
query_groups:
- queries: [sum(metric)]
repetition_delay_ms: 60000
controller_options:
accuracy_sla: 0.99
"#;

let config: ControllerConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(
config.query_groups[0].controller_options.latency_sla_ms,
None
);
}

// Reject the unitless `latency_sla` key so it can't be silently ignored.
#[test]
fn rejects_legacy_latency_sla_key() {
let yaml = r#"
query_groups:
- queries: [sum(metric)]
repetition_delay_ms: 60000
controller_options:
accuracy_sla: 0.99
latency_sla: 1.0
"#;

let error = serde_yaml::from_str::<ControllerConfig>(yaml)
.expect_err("legacy latency_sla key must be rejected")
.to_string();
assert!(error.contains("unknown field `latency_sla`"), "{error}");
}

#[test]
fn rejects_misspelled_controller_options() {
// A typo would otherwise drop the whole SLA block to its defaults.
let yaml = r#"
query_groups:
- queries: [sum(metric)]
repetition_delay_ms: 60000
controler_options:
accuracy_sla: 0.99
"#;

let error = serde_yaml::from_str::<ControllerConfig>(yaml)
.expect_err("unknown query group key must be rejected")
.to_string();
assert!(
error.contains("unknown field `controler_options`"),
"{error}"
);
}

#[test]
fn sql_and_elastic_configs_reject_unknown_group_keys() {
let sql = r#"
tables: []
query_groups:
- queries: ["SELECT 1"]
repetition_delay_ms: 60000
latency_sla: 1.0
controller_options:
accuracy_sla: 0.99
"#;
let error = serde_yaml::from_str::<SQLControllerConfig>(sql)
.expect_err("unknown SQL group key must be rejected")
.to_string();
assert!(error.contains("unknown field `latency_sla`"), "{error}");

let elastic = r#"
query_groups:
- queries: ["{}"]
repetition_delay_ms: 60000
index: i
time_field: t
controler_options:
accuracy_sla: 0.99
"#;
let error = serde_yaml::from_str::<ElasticDSLControllerConfig>(elastic)
.expect_err("unknown Elastic group key must be rejected")
.to_string();
assert!(
error.contains("unknown field `controler_options`"),
"{error}"
);
}

#[test]
fn rejects_non_positive_or_non_finite_latency_sla_ms() {
for bad in ["0.0", "-5.0", ".inf", ".nan"] {
let yaml = format!(
r#"
query_groups:
- queries: [sum(metric)]
repetition_delay_ms: 60000
controller_options:
accuracy_sla: 0.99
latency_sla_ms: {bad}
"#
);

let error = serde_yaml::from_str::<ControllerConfig>(&yaml)
.expect_err("invalid latency_sla_ms must be rejected")
.to_string();
assert!(
error.contains("must be a finite number greater than zero"),
"{bad}: {error}"
);
}
}

#[test]
Expand Down
26 changes: 20 additions & 6 deletions asap-planner-rs/src/optimizer/aqe_extractor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ pub struct RQE {
pub query_string: String,
pub t_repeat_ms: u64,
pub accuracy_sla: f64,
pub latency_sla: f64,
pub latency_sla_ms: Option<f64>,
}

/// Stable key for merging identical optimizer demand.
Expand All @@ -36,7 +36,7 @@ struct OptimizerItemKey {
topk_by_labels: Option<KeyByLabelNames>,
t_repeat_ms: u64,
accuracy_sla_bits: u64,
latency_sla_bits: u64,
latency_sla_ms_bits: Option<u64>,
}

impl OptimizerItemKey {
Expand All @@ -51,7 +51,7 @@ impl OptimizerItemKey {
topk_by_labels: req.topk_by_labels.clone(),
t_repeat_ms: rqe.t_repeat_ms,
accuracy_sla_bits: normalized_f64_bits(rqe.accuracy_sla),
latency_sla_bits: normalized_f64_bits(rqe.latency_sla),
latency_sla_ms_bits: rqe.latency_sla_ms.map(normalized_f64_bits),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The positive and finite rule for latency_sla_ms is enforced only in the serde deserializer. RQEs built in code (tests, query_log conversion, future callers) can pass Some(0.0), Some(-0.0) or Some(NaN) into the key and the MILP latency ceiling. NaN also gives otherwise-equal keys different bits. Consider validating where the RQE is built, or using a PositiveMs newtype.

Also, no test checks that items with different latency_sla_ms values (or Some vs None) stay distinct. different_slas_become_distinct_items only varies accuracy_sla, and the signed-zero test was moved to accuracy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added different_latency_slas_become_distinct_items in 0d3f0a8: None, Some(100) and Some(1000) stay three items. Skipping the newtype for now: every non-test RQE is built from a deserialized ControllerOptions (pipeline.rs, candidate_gen_dump.rs), and query_log uses Default (None), so Some(0.0) / NaN cannot reach the key today. We can add a PositiveMs newtype if a code-built RQE path appears.

}
}
}
Expand Down Expand Up @@ -110,7 +110,7 @@ pub fn extract_aqes(
query_frequency_hz,
t_repeat_ms: key.t_repeat_ms,
accuracy_sla: f64::from_bits(key.accuracy_sla_bits),
latency_sla: f64::from_bits(key.latency_sla_bits),
latency_sla_ms: key.latency_sla_ms_bits.map(f64::from_bits),
},
)
.collect())
Expand Down Expand Up @@ -236,7 +236,7 @@ mod tests {
query_string: query.to_string(),
t_repeat_ms: t_ms,
accuracy_sla: 0.0,
latency_sla: 0.0,
latency_sla_ms: None,
}
}

Expand Down Expand Up @@ -309,10 +309,24 @@ mod tests {
assert_eq!(items.len(), 2);
}

#[test]
fn different_latency_slas_become_distinct_items() {
let unlimited = rqe("sum_over_time(metric[5m])", 60_000);
let mut fast = rqe("sum_over_time(metric[5m])", 60_000);
fast.latency_sla_ms = Some(100.0);
let mut slow = rqe("sum_over_time(metric[5m])", 60_000);
slow.latency_sla_ms = Some(1_000.0);

let items = extract_aqes(&[unlimited, fast, slow], &empty_schema(), 15_000).unwrap();
let mut slas: Vec<_> = items.iter().map(|item| item.latency_sla_ms).collect();
slas.sort_by(|a, b| a.partial_cmp(b).unwrap());
assert_eq!(slas, vec![None, Some(100.0), Some(1_000.0)]);
}

#[test]
fn signed_zero_slas_merge_into_one_item() {
let mut negative_zero = rqe("sum_over_time(metric[5m])", 60_000);
negative_zero.latency_sla = -0.0;
negative_zero.accuracy_sla = -0.0;
let items = extract_aqes(
&[negative_zero, rqe("sum_over_time(metric[5m])", 60_000)],
&empty_schema(),
Expand Down
2 changes: 1 addition & 1 deletion asap-planner-rs/src/optimizer/candidate_gen.rs
Original file line number Diff line number Diff line change
Expand Up @@ -408,7 +408,7 @@ mod tests {
query_frequency_hz: 1.0 / 60.0,
t_repeat_ms: min_t,
accuracy_sla: 0.0,
latency_sla: 0.0,
latency_sla_ms: None,
}
}

Expand Down
2 changes: 1 addition & 1 deletion asap-planner-rs/src/optimizer/cost_model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ mod tests {
query_frequency_hz: 1.0 / 60.0,
t_repeat_ms: min_t,
accuracy_sla: 0.0,
latency_sla: 0.0,
latency_sla_ms: None,
}
}

Expand Down
2 changes: 1 addition & 1 deletion asap-planner-rs/src/optimizer/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ pub struct UnservableItem {
pub data_range_ms: u64,
pub t_repeat_ms: u64,
pub accuracy_sla: f64,
pub latency_sla: f64,
pub latency_sla_ms: Option<f64>,
pub reason: String,
}
6 changes: 3 additions & 3 deletions asap-planner-rs/src/optimizer/greedy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ pub fn greedy_assign(
data_range_ms: aqe.requirements.data_range_ms,
t_repeat_ms: aqe.t_repeat_ms,
accuracy_sla: aqe.accuracy_sla,
latency_sla: aqe.latency_sla,
latency_sla_ms: aqe.latency_sla_ms,
reason: "no candidate remained after structural and atomic-cost filters".into(),
});
continue;
Expand Down Expand Up @@ -139,7 +139,7 @@ mod tests {
query_frequency_hz: freq_hz,
t_repeat_ms: min_t,
accuracy_sla: 0.0,
latency_sla: 0.0,
latency_sla_ms: None,
}
}

Expand Down Expand Up @@ -196,7 +196,7 @@ mod tests {
query_frequency_hz: 1.0 / 60.0,
t_repeat_ms: 60_000,
accuracy_sla: 0.0,
latency_sla: 0.0,
latency_sla_ms: None,
};
let error = greedy_assign(
vec![aqe.clone()],
Expand Down
2 changes: 1 addition & 1 deletion asap-planner-rs/src/optimizer/label_set_facts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ mod tests {
query_frequency_hz: 1.0 / 60.0,
t_repeat_ms: 60_000,
accuracy_sla: 0.0,
latency_sla: 0.0,
latency_sla_ms: None,
}
}

Expand Down
2 changes: 1 addition & 1 deletion asap-planner-rs/src/optimizer/pipeline.rs
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ fn config_to_rqes(config: &ControllerConfig) -> Vec<RQE> {
query_string: q.clone(),
t_repeat_ms: qg.repetition_delay_ms,
accuracy_sla: qg.controller_options.accuracy_sla,
latency_sla: qg.controller_options.latency_sla,
latency_sla_ms: qg.controller_options.latency_sla_ms,
})
})
.collect()
Expand Down
Loading
Loading