Reach kolla-ansible's own playbooks, and keep its fact cache fresh - #2670
Draft
ideaship wants to merge 2 commits into
Draft
Reach kolla-ansible's own playbooks, and keep its fact cache fresh#2670ideaship wants to merge 2 commits into
ideaship wants to merge 2 commits into
Conversation
The kolla branch of Run._prepare_task lets osism-ansible claim a role
whenever osism-ansible ships a playbook of that name:
role not in ["common"]
and "osism-ansible" in MAP_ROLE2RUNTIME
and role in MAP_ROLE2RUNTIME["osism-ansible"]
The override is needed: several kolla-environment playbooks live in
osism-ansible rather than kolla-ansible (ansible-playbooks
playbooks/kolla/: stepca, fix-gh973, nova-update-cell-mappings,
openvswitch-ipfix) and must run in that runtime. But the test only asks
whether osism-ansible ships the name, never which environment it ships
it for, and the kolla- prefix is stripped just above. Any name
kolla-ansible advertises as kolla-<x> whose bare form osism-ansible
registers for a different environment therefore collapses onto that
entry and is dispatched to osism-ansible with ENVIRONMENT=kolla, where
run-kolla.sh looks for /ansible/kolla/<x>.yml, finds nothing and reports
ERROR: service <x> in environment kolla not available
Exactly three names collide, all registered by osism-ansible for the
generic environment: facts, gather-facts and certificates. So
osism apply kolla-facts, kolla-gather-facts and kolla-certificates have
all been unreachable, in every spelling, since #877 introduced the
override -- kolla-certificates from two days after #612 deliberately
renamed the play to make it addressable, and kolla-gather-facts from
2024-06-26, when generic/gather-facts.yml was added. Nobody noticed
because all three were redundant at the time: a single fact-cache
keyspace meant osism-ansible's gather served kolla too, and OSISM
handles TLS separately from kolla's certificates play.
Require the role to be mapped to the kolla environment as well. stepca
and friends map to kolla and keep running in osism-ansible; facts,
gather-facts and certificates map to generic, fail the guard and fall
through to kolla-ansible, where those playbooks actually live.
This matters now because ansible-core 2.19 namespaces fact-cache
entries per schema, so kolla-ansible needs to write its own generation
rather than read osism-ansible's, and kolla-facts is how it does that.
The fix itself is version independent: these commands should work on any
ansible-core.
Verified against the interface maps of a live cluster: facts,
kolla-facts, gather-facts, certificates and kolla-certificates now route
to kolla-ansible, while stepca, fix-gh973 and
nova-update-cell-mappings still route to osism-ansible, and keystone and
common are unchanged.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
Since ansible-core 2.19 the fact cache is namespaced per schema, so
osism-ansible's gather no longer fills the keyspace kolla-ansible reads
when the two straddle 2.19. kolla never gathers for itself -- the site
splitter forces gather_facts: false on every emitted play -- so its
generation has no writer at all.
Give it one. dispatch_kolla_facts() queues /ansible/kolla-facts.yml,
OSISM's own unconditional kolla-side gather, in the kolla-ansible
runtime. The periodic gather_facts task calls it after the generic
gather, and osism sync facts does the same so a single operator command
repairs both generations; check_ansible_facts already points operators
at that command, and until now its advice could not fix the kolla case.
Without the periodic half a one-shot gather decays: fact_caching_timeout
is 86400 while GATHER_FACTS_SCHEDULE is 43200, and only osism-ansible's
generation was ever rewritten, so kolla's would expire a day after a
successful deploy with no deployment in flight to explain the failure.
The dispatch is guarded on kolla-ansible appearing in MAP_ROLE2RUNTIME.
The kolla-ansible container is the only consumer of its own Celery queue
and is gated on enable_kolla_ansible, so on a manager without it
task_create_missing_queues would create the queue and every run would
strand another task in Redis.
The guard has to run in a worker rather than in setup_periodic_tasks:
/interface is not mounted into the beat container, so MAP_ROLE2RUNTIME
is empty there and the guard would never pass. Verified on a live
cluster -- in a container without /interface the map is {} and the
dispatch is skipped, while osismclient sees ceph-ansible, kolla-ansible
and osism-ansible.
sync facts returns early if the generic gather fails, so a broken
osism-ansible run is reported as-is instead of being masked by the kolla
result.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The kolla branch of
Run._prepare_tasklets osism-ansible claim a role wheneverosism-ansible ships a playbook of that name:
The override is needed —
stepca,fix-gh973,nova-update-cell-mappingsandfriends are kolla-environment playbooks implemented in osism-ansible. But the
test never asks which environment osism-ansible ships the name for, and the
kolla-prefix is stripped just above. So any name kolla-ansible advertises askolla-<x>whose bare form osism-ansible registers for a different environmentcollapses onto that entry and is sent to osism-ansible with
ENVIRONMENT=kolla,where
run-kolla.shfinds nothing:Exactly three names collide, all registered by osism-ansible for
generic:osism apply kolla-factsosism apply kolla-gather-factsosism apply kolla-certificatesAll three have been unreachable, in every spelling, since #877 added the
override —
kolla-certificatesfrom two days after #612 deliberately renamed theplay to make it addressable. Nobody noticed because all three were redundant
then: one fact-cache keyspace meant osism-ansible's gather served kolla too, and
OSISM handles TLS separately from kolla's
certificatesplay.ansible-core 2.19 changed that by namespacing cache entries per schema, so kolla
now needs to write its own generation. Background:
What
apply: keep kolla playbooks in kolla-ansible— also requireMAP_ROLE2ENVIRONMENT.get(role) == "kolla".stepcaand friends map tokollaand keep running in osism-ansible;facts,gather-factsandcertificatesmap togenericand fall through to kolla-ansible. Versionindependent: these commands should work on any ansible-core.
facts: gather in the kolla runtime too—dispatch_kolla_facts()queues/ansible/kolla-facts.ymlin the kolla-ansible runtime; the periodicgather_factstask andosism sync factsboth call it. Without the periodichalf a one-shot gather decays:
fact_caching_timeoutis 86400 whileGATHER_FACTS_SCHEDULEis 43200, and only osism-ansible's generation was everrewritten. Guarded on
kolla-ansiblebeing inMAP_ROLE2RUNTIME, since thatcontainer is the only consumer of its own queue and is
enable_kolla_ansible-gated.The guard runs in a worker, not in
setup_periodic_tasks:/interfaceis notmounted into the beat container, so
MAP_ROLE2RUNTIMEis empty there and theguard would never pass. Verified live — in a container without
/interfacethemap is
{}and the dispatch is skipped;osismclientsees all three runtimes.Verification
3245 unit tests pass; 8 new ones cover both routing directions, both guard
branches and the
sync factspaths.Live routing probe against a real cluster's
/interface/playbooksmaps:End-to-end:
osism apply kolla-factsruns 7/7 hostsfailed=0and resets theRedis TTL, with no manual
-eflags.osism sync factsproduces two PLAYRECAPs, one per runtime.
Companion
The playbook-side half — the
gather_subsetfix that makeskolla-facts.ymlrunnable, and the gather import that covers the deploy path — is in:
That PR is what fixes the deploy path; this one makes the operator commands and
the periodic refresh work. Land the
gather_subsetfix first, since thesecommands otherwise reach a playbook that cannot run.
🤖 Generated with Claude Code