Skip to content

Reach kolla-ansible's own playbooks, and keep its fact cache fresh - #2670

Draft
ideaship wants to merge 2 commits into
mainfrom
fix-kolla-environment-routing
Draft

Reach kolla-ansible's own playbooks, and keep its fact cache fresh#2670
ideaship wants to merge 2 commits into
mainfrom
fix-kolla-environment-routing

Conversation

@ideaship

@ideaship ideaship commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

The kolla branch of Run._prepare_task lets osism-ansible claim a role whenever
osism-ansible ships a playbook of that name:

role not in ["common"]
and "osism-ansible" in MAP_ROLE2RUNTIME
and role in MAP_ROLE2RUNTIME["osism-ansible"]

The override is needed — stepca, fix-gh973, nova-update-cell-mappings and
friends are kolla-environment playbooks implemented in osism-ansible. But the
test never asks which environment osism-ansible ships the name for, and the
kolla- prefix is stripped just above. So any name kolla-ansible advertises as
kolla-<x> whose bare form osism-ansible registers for a different environment
collapses onto that entry and is sent to osism-ansible with ENVIRONMENT=kolla,
where run-kolla.sh finds nothing:

ERROR: service <x> in environment kolla not available

Exactly three names collide, all registered by osism-ansible for generic:

command before after
osism apply kolla-facts not available runs
osism apply kolla-gather-facts not available runs
osism apply kolla-certificates not available runs

All three have been unreachable, in every spelling, since #877 added the
override — kolla-certificates from two days after #612 deliberately renamed the
play to make it addressable. Nobody noticed because all three were redundant
then: one fact-cache keyspace meant osism-ansible's gather served kolla too, and
OSISM handles TLS separately from kolla's certificates play.

ansible-core 2.19 changed that by namespacing cache entries per schema, so kolla
now needs to write its own generation. Background:

What

  1. apply: keep kolla playbooks in kolla-ansible — also require
    MAP_ROLE2ENVIRONMENT.get(role) == "kolla". stepca and friends map to
    kolla and keep running in osism-ansible; facts, gather-facts and
    certificates map to generic and fall through to kolla-ansible. Version
    independent: these commands should work on any ansible-core.
  2. facts: gather in the kolla runtime toodispatch_kolla_facts() queues
    /ansible/kolla-facts.yml in the kolla-ansible runtime; the periodic
    gather_facts task and osism sync facts both call it. Without the periodic
    half a one-shot gather decays: fact_caching_timeout is 86400 while
    GATHER_FACTS_SCHEDULE is 43200, and only osism-ansible's generation was ever
    rewritten. Guarded on kolla-ansible being in MAP_ROLE2RUNTIME, since that
    container is the only consumer of its own queue and is
    enable_kolla_ansible-gated.

The guard runs in a worker, not in setup_periodic_tasks: /interface is not
mounted into the beat container, so MAP_ROLE2RUNTIME is empty there and the
guard would never pass. Verified live — in a container without /interface the
map is {} and the dispatch is skipped; osismclient sees all three runtimes.

Verification

3245 unit tests pass; 8 new ones cover both routing directions, both guard
branches and the sync facts paths.

Live routing probe against a real cluster's /interface/playbooks maps:

facts, kolla-facts, gather-facts,
certificates, kolla-certificates      -> kolla-ansible
stepca, fix-gh973,
nova-update-cell-mappings             -> osism-ansible
keystone, common                      -> kolla-ansible   (unchanged)

End-to-end: osism apply kolla-facts runs 7/7 hosts failed=0 and resets the
Redis TTL, with no manual -e flags. osism sync facts produces two PLAY
RECAPs, one per runtime.

Companion

The playbook-side half — the gather_subset fix that makes kolla-facts.yml
runnable, and the gather import that covers the deploy path — is in:

That PR is what fixes the deploy path; this one makes the operator commands and
the periodic refresh work. Land the gather_subset fix first, since these
commands otherwise reach a playbook that cannot run.

🤖 Generated with Claude Code

The kolla branch of Run._prepare_task lets osism-ansible claim a role
whenever osism-ansible ships a playbook of that name:

    role not in ["common"]
    and "osism-ansible" in MAP_ROLE2RUNTIME
    and role in MAP_ROLE2RUNTIME["osism-ansible"]

The override is needed: several kolla-environment playbooks live in
osism-ansible rather than kolla-ansible (ansible-playbooks
playbooks/kolla/: stepca, fix-gh973, nova-update-cell-mappings,
openvswitch-ipfix) and must run in that runtime. But the test only asks
whether osism-ansible ships the name, never which environment it ships
it for, and the kolla- prefix is stripped just above. Any name
kolla-ansible advertises as kolla-<x> whose bare form osism-ansible
registers for a different environment therefore collapses onto that
entry and is dispatched to osism-ansible with ENVIRONMENT=kolla, where
run-kolla.sh looks for /ansible/kolla/<x>.yml, finds nothing and reports

    ERROR: service <x> in environment kolla not available

Exactly three names collide, all registered by osism-ansible for the
generic environment: facts, gather-facts and certificates. So
osism apply kolla-facts, kolla-gather-facts and kolla-certificates have
all been unreachable, in every spelling, since #877 introduced the
override -- kolla-certificates from two days after #612 deliberately
renamed the play to make it addressable, and kolla-gather-facts from
2024-06-26, when generic/gather-facts.yml was added. Nobody noticed
because all three were redundant at the time: a single fact-cache
keyspace meant osism-ansible's gather served kolla too, and OSISM
handles TLS separately from kolla's certificates play.

Require the role to be mapped to the kolla environment as well. stepca
and friends map to kolla and keep running in osism-ansible; facts,
gather-facts and certificates map to generic, fail the guard and fall
through to kolla-ansible, where those playbooks actually live.

This matters now because ansible-core 2.19 namespaces fact-cache
entries per schema, so kolla-ansible needs to write its own generation
rather than read osism-ansible's, and kolla-facts is how it does that.
The fix itself is version independent: these commands should work on any
ansible-core.

Verified against the interface maps of a live cluster: facts,
kolla-facts, gather-facts, certificates and kolla-certificates now route
to kolla-ansible, while stepca, fix-gh973 and
nova-update-cell-mappings still route to osism-ansible, and keystone and
common are unchanged.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Since ansible-core 2.19 the fact cache is namespaced per schema, so
osism-ansible's gather no longer fills the keyspace kolla-ansible reads
when the two straddle 2.19. kolla never gathers for itself -- the site
splitter forces gather_facts: false on every emitted play -- so its
generation has no writer at all.

Give it one. dispatch_kolla_facts() queues /ansible/kolla-facts.yml,
OSISM's own unconditional kolla-side gather, in the kolla-ansible
runtime. The periodic gather_facts task calls it after the generic
gather, and osism sync facts does the same so a single operator command
repairs both generations; check_ansible_facts already points operators
at that command, and until now its advice could not fix the kolla case.

Without the periodic half a one-shot gather decays: fact_caching_timeout
is 86400 while GATHER_FACTS_SCHEDULE is 43200, and only osism-ansible's
generation was ever rewritten, so kolla's would expire a day after a
successful deploy with no deployment in flight to explain the failure.

The dispatch is guarded on kolla-ansible appearing in MAP_ROLE2RUNTIME.
The kolla-ansible container is the only consumer of its own Celery queue
and is gated on enable_kolla_ansible, so on a manager without it
task_create_missing_queues would create the queue and every run would
strand another task in Redis.

The guard has to run in a worker rather than in setup_periodic_tasks:
/interface is not mounted into the beat container, so MAP_ROLE2RUNTIME
is empty there and the guard would never pass. Verified on a live
cluster -- in a container without /interface the map is {} and the
dispatch is skipped, while osismclient sees ceph-ansible, kolla-ansible
and osism-ansible.

sync facts returns early if the generic gather fails, so a broken
osism-ansible run is reported as-is instead of being masked by the kolla
result.

Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

2 participants