Skip to content

Add CIMC IPMI stall and APIC identity check (CSCwe61589) - #498

Open
tskanai1 wants to merge 1 commit into
datacenter:v4.3.1-devfrom
tskanai1:cscwe61589-cimc-check
Open

tskanai1 wants to merge 1 commit into
datacenter:v4.3.1-devfrom
tskanai1:cscwe61589-cimc-check

Conversation

@tskanai1

@tskanai1 tskanai1 commented Nov 17, 2026 •

Copy link
Copy Markdown

Closes #491

What this adds

A new check, cimc_cscwe61589_check, for CSCwe61589 ("I2C Controller 5 software FIFO is full"). When the CIMC IPMI stack of an APIC gets into a restart loop, the APIC can read an incorrect product name and serial number at boot. It then generates its certificate from that identity and cannot rejoin the cluster during an upgrade, so the cluster upgrade gets stuck and Cisco TAC support is needed to complete it.

For every APIC the check evaluates two things:

Condition Source Result
CIMC version is affected (APIC M3/L3 and M4/L4): 4.0/4.1, 4.2 older than 4.2(3g), 4.3(1.x), 4.3(2.x) older than build 230207, 6.0(1.x) older than build 250127 eqptCh.cimcVersion MANUAL CHECK REQUIRED
APIC identity from acidiag show cimc is incorrect (empty value, default serial, PID used as serial, product name not an APIC model, PID differs from product name) and the CIMC version is affected SSH acidiag show cimc (works as admin) FAIL - UPGRADE FAILURE!!
Identity cannot be retrieved, serial differs from eqptCh, or incorrect identity without an affected CIMC same MANUAL CHECK REQUIRED
acidiag show cimc does not exist on the release PASS; only the CIMC version is checked

Notes for review

  • The CIMC version helper is_firstver_gt_secondver compares single characters and cannot handle build-style versions such as 4.3(2.250016), so the check has its own parser (cimc_cscwe61589_is_affected).
  • The check uses SSH, so it is registered with the connection-based checks and is skipped with the API-only option.
  • ipmitool and dmidecode need root, so a CIMC that is currently in the restart loop cannot be detected directly as admin. The docs describe how to confirm it from the CIMC technical support.
  • APIC M4/L4 (UCS C225 M6) are evaluated because the defect record covers M6 racks and those APICs support the overlapping CIMC builds. C225-specific applicability is not independently confirmed.
  • The fixed versions come from the defect record. 4.3(2.230207) was confirmed as a fixed build.

Testing

  • Full suite: 1502 passed on Python 3.11, including 43 new tests that cover data, logic and results only.
  • acidiag show cimc was run read-only as admin on lab APICs running 6.1(6g) and 6.2(3g), and its output format is covered by the tests. Its absence was confirmed on a 6.0(3d) APIC, and that real output is used as the unsupported-release fixture.
  • The script was run end to end on lab APICs as admin (results in the PR comment). An APIC with an incorrect identity is not available in the lab, so that path is covered by unit tests only.
  • Docs are in the Defect Check Details section of docs/docs/validations.md.

When the CIMC IPMI stack of an APIC gets into a restart loop (CSCwe61589),
the APIC can read an incorrect product name and serial number at boot,
generate its certificate from them, and fail to rejoin the cluster during
an upgrade.

The new cimc_cscwe61589_check evaluates every APIC for:
- CIMC version: affected versions on APIC M3/L3 and M4/L4 return
  MANUAL CHECK REQUIRED. Fixed versions are 4.2(3g), 4.3(2.230207) and
  later. The check parses both CIMC version formats because the existing
  helper cannot compare build-style versions.
- APIC identity from `acidiag show cimc`, which works as the admin user:
  an incorrect identity together with an affected CIMC version returns
  FAIL - UPGRADE FAILURE. An identity that cannot be retrieved returns
  MANUAL CHECK REQUIRED. Releases without the command only get the CIMC
  version check.

The check uses SSH, so it is registered with the connection-based checks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tskanai1

Copy link
Copy Markdown
Author

End-to-end test results

Commit 4b6d72b was run on lab APICs as admin with -d cimc_cscwe61589_check -t '6.2(3g)' (this check only).

APIC ACI version CIMC versions Result
APIC-SERVER-M4 x3 6.1(6g) 4.3(4.241063) x2, 4.2(3b) x1 MANUAL CHECK REQUIRED, one row for the APIC with CIMC 4.2(3b)
APIC-SERVER-M4 x3 6.1(5e) 4.3(4.241063) x3 PASS
APIC-SERVER-M4 x3 6.1(4h) 4.3(4.252002) x3 PASS
APIC-SERVER-L3 x3 6.2(3g) 4.3(2.250016) x3 PASS
APIC-SERVER-L3 x3 6.0(9d) 4.1(1g) x2, 4.1(3c) x1 MANUAL CHECK REQUIRED, one row per APIC. acidiag show cimc is not available on this release, so only the CIMC version was evaluated
APIC-SERVER-L4 x3 6.0(3d) 4.3(4.241063) x2, 6.0(1.250127) x1 PASS. acidiag show cimc is not available on this release, so only the CIMC version was evaluated

Where acidiag show cimc is available, it ran through the script's own SSH connection on every APIC and returned the identity details with no findings. Not covered: an APIC with an incorrect identity.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant