Repository navigation
Conversation
When the CIMC IPMI stack of an APIC gets into a restart loop (CSCwe61589), the APIC can read an incorrect product name and serial number at boot, generate its certificate from them, and fail to rejoin the cluster during an upgrade. The new cimc_cscwe61589_check evaluates every APIC for: - CIMC version: affected versions on APIC M3/L3 and M4/L4 return MANUAL CHECK REQUIRED. Fixed versions are 4.2(3g), 4.3(2.230207) and later. The check parses both CIMC version formats because the existing helper cannot compare build-style versions. - APIC identity from `acidiag show cimc`, which works as the admin user: an incorrect identity together with an affected CIMC version returns FAIL - UPGRADE FAILURE. An identity that cannot be retrieved returns MANUAL CHECK REQUIRED. Releases without the command only get the CIMC version check. The check uses SSH, so it is registered with the connection-based checks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Author
End-to-end test resultsCommit 4b6d72b was run on lab APICs as
Where |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #491
What this adds
A new check,
cimc_cscwe61589_check, for CSCwe61589 ("I2C Controller 5 software FIFO is full"). When the CIMC IPMI stack of an APIC gets into a restart loop, the APIC can read an incorrect product name and serial number at boot. It then generates its certificate from that identity and cannot rejoin the cluster during an upgrade, so the cluster upgrade gets stuck and Cisco TAC support is needed to complete it.For every APIC the check evaluates two things:
eqptCh.cimcVersionMANUAL CHECK REQUIREDacidiag show cimcis incorrect (empty value, default serial, PID used as serial, product name not an APIC model, PID differs from product name) and the CIMC version is affectedacidiag show cimc(works asadmin)FAIL - UPGRADE FAILURE!!eqptCh, or incorrect identity without an affected CIMCMANUAL CHECK REQUIREDacidiag show cimcdoes not exist on the releasePASS; only the CIMC version is checkedNotes for review
is_firstver_gt_secondvercompares single characters and cannot handle build-style versions such as4.3(2.250016), so the check has its own parser (cimc_cscwe61589_is_affected).ipmitoolanddmidecodeneed root, so a CIMC that is currently in the restart loop cannot be detected directly asadmin. The docs describe how to confirm it from the CIMC technical support.Testing
acidiag show cimcwas run read-only asadminon lab APICs running 6.1(6g) and 6.2(3g), and its output format is covered by the tests. Its absence was confirmed on a 6.0(3d) APIC, and that real output is used as the unsupported-release fixture.admin(results in the PR comment). An APIC with an incorrect identity is not available in the lab, so that path is covered by unit tests only.docs/docs/validations.md.