You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Brief description in the context of pre-upgrade validation: the CIMC of an APIC server hits an application thread stall, and IPMI (cipmi / doctor-bmc) gets into a constant restart loop. Typical CIMC log lines:
cipmi: message.c: Unable to add packet to app tx list
cipmi: oem_common.c: Error: the 'application' thread is stalled.
doctor-bmc: doctor-bmc.c: Restarting IPMI Stack->Reason: VKCS UnResponsive
doctor-bmc: doctor-bmc.c: Sensor scan file is not updated.
kernel: I2C Controller 5 software FIFO is full
Known fixed releases (per the bug): 4.2(3g)
Workaround: restart the CIMC
Proposed check, for each APIC before the upgrade:
Collect the CIMC firmware version and flag it if it is below 4.2(3g).
Flag the APIC if CSCwe61589 is currently occurring (IPMI/CIMC unresponsive or in a restart loop), and tell the user to restart the CIMC first, before rebooting the APIC or starting/continuing the upgrade.
If possible, also verify that the PID and serial number the APIC reports are sane (not empty / not unexpected), see "Why it needs to be validated".
Why it needs to be validated
An upgrade recently failed on a single APIC node because of this bug. If CSCwe61589 is occurring and the workaround (CIMC restart) is not performed before the APIC is rebooted:
At boot, the APIC obtains an incorrect PID and serial number from the hardware.
If the upgrade is started in this state, the APIC regenerates its certificate based on this incorrect information.
The APIC becomes unreachable and the upgrade cannot be completed.
The certificate has to be recreated, which requires a reimage of the affected APIC, and the in-progress upgrade has to be aborted.
Restarting the CIMC before rebooting the APIC, or upgrading the CIMC to a fixed version before starting the upgrade, avoids the problem. The occurrence of CSCwe61589 has to be detected before the upgrade.
Additional context
Example of what was observed (details omitted): during an upgrade of a 3-node APIC cluster, APIC-2 became abnormal and was no longer visible from the other APICs, so the cluster upgrade could not be completed. On APIC-2, the PID and serial number differed from the actual hardware values, and the certificate had been generated incorrectly based on them. At that point, APIC-2 had already moved to the target version while APIC-3 was still waiting for its turn at the data conversion. The upgrade had to be aborted: APIC-3 and APIC-1 were returned to the original version, and APIC-2 was decommissioned, reimaged and commissioned back. This pre-upgrade validation script had been run before the upgrade and did not flag anything related to this.
This script can query APIC APIs and execute user-level commands on each APIC.
Please provide the exact query/command evidence that supports that we can actually check for symptoms associated with this behavior when queried from the APIC via user-level access.
Please confirm the logs attached. Sensitive information has been masked.
If the APIC is running a CIMC version earlier than the version that includes the fix for CSCwe61589, it is important to verify the serial number using the acidiag show cimc command.
acidiag show cimc
==================== CIMC DETAILS ====================
PID : APIC-SERVER-L3
Serial Number : UCSC-C220-M5 <<<
Product Name : UCSC-C220-M5
The following logs were collected after the upgrade failed.
moquery -c faultInst -f 'fault.Inst.highestSeverity=="critical"'
Total Objects shown: 1
fault.Inst
code : F3031
ack : no
alert : no
annotation :
cause : cert-invalid
changeSet :
childAction :
created : 2026-09-08T18:03:54.958+09:00
delegated : no
descr : Node Certificate is invalid: Failed to parse the subject line as a valid ACI fabric certificate AND Invalid Serial Number AND Invalid Product ID
dn : uni/fabsslcomm/ifmcertnode-2/fault-F3031
domain : security
extMngdBy : undefined
highestSeverity : critical
lastTransition : 2026-09-08T18:05:59.218+09:00
lc : raised
modTs : never
occur : 1
origSeverity : critical
prevSeverity : critical
rn : fault-F3031
rule : pki-fabric-nodesslcertificate-node-cert-invalid
severity : critical
status :
subject : security-provider
title :
type : operational
uid :
userdom : all
5057||2026-09-07T12:19:27.450857038+00:00||proc_app||ERROR||||Failed to find matching system-product-name||../common/src/ifcutils/ifc_pid_check.cc||104
Activity
monrog2 commentedon Nov 12, 2026
This script can query APIC APIs and execute user-level commands on each APIC.
Please provide the exact query/command evidence that supports that we can actually check for symptoms associated with this behavior when queried from the APIC via user-level access.
mari9825 commentedon Nov 16, 2026
Please confirm the logs attached. Sensitive information has been masked.
If the APIC is running a CIMC version earlier than the version that includes the fix for CSCwe61589, it is important to verify the serial number using the acidiag show cimc command.
acidiag show cimc
The following logs were collected after the upgrade failed.
dbgexp_tsod-techsupport_sysid-2_2026-09-08T18-57JST_logs_3of3/data/log/svc_ifc_appliancedirector.bin.warnplus.log
acidiag verify apic
dbgexp_tsod-techsupport_sysid-2_2026-09-08T18-57JST_1of3/var/log/dme/oldlog/cli[LOG_ID]/verify_apic
tskanai1 commentedon Nov 17, 2026
@monrog2 @mari9825 I implemented the new check against CSCwe61589 and opened the PR #498.