Skip to content

NewValidation: [Bug] CSCwe61589 - APIC CIMC IPMI restart loop ("I2C Controller 5 software FIFO is full") #491

Open

Description

@mari9825

(use upvote 👍 for attentions)

Validation Type

[ ] - Fault

[ ] - Config

[x] - Bug

[ ] - Other

What needs to be validated

Bug ID: CSCwe61589 - "I2C Controller 5 software FIFO is full"

Brief description in the context of pre-upgrade validation: the CIMC of an APIC server hits an application thread stall, and IPMI (cipmi / doctor-bmc) gets into a constant restart loop. Typical CIMC log lines:

cipmi: message.c: Unable to add packet to app tx list
cipmi: oem_common.c: Error: the 'application' thread is stalled.
doctor-bmc: doctor-bmc.c: Restarting IPMI Stack->Reason: VKCS UnResponsive
doctor-bmc: doctor-bmc.c: Sensor scan file is not updated.
kernel: I2C Controller 5 software FIFO is full
  • Known fixed releases (per the bug): 4.2(3g)
  • Workaround: restart the CIMC

Proposed check, for each APIC before the upgrade:

  1. Collect the CIMC firmware version and flag it if it is below 4.2(3g).
  2. Flag the APIC if CSCwe61589 is currently occurring (IPMI/CIMC unresponsive or in a restart loop), and tell the user to restart the CIMC first, before rebooting the APIC or starting/continuing the upgrade.
  3. If possible, also verify that the PID and serial number the APIC reports are sane (not empty / not unexpected), see "Why it needs to be validated".

Why it needs to be validated

An upgrade recently failed on a single APIC node because of this bug. If CSCwe61589 is occurring and the workaround (CIMC restart) is not performed before the APIC is rebooted:

  1. At boot, the APIC obtains an incorrect PID and serial number from the hardware.
  2. If the upgrade is started in this state, the APIC regenerates its certificate based on this incorrect information.
  3. The APIC becomes unreachable and the upgrade cannot be completed.
  4. The certificate has to be recreated, which requires a reimage of the affected APIC, and the in-progress upgrade has to be aborted.

Restarting the CIMC before rebooting the APIC, or upgrading the CIMC to a fixed version before starting the upgrade, avoids the problem. The occurrence of CSCwe61589 has to be detected before the upgrade.

Additional context

Example of what was observed (details omitted): during an upgrade of a 3-node APIC cluster, APIC-2 became abnormal and was no longer visible from the other APICs, so the cluster upgrade could not be completed. On APIC-2, the PID and serial number differed from the actual hardware values, and the certificate had been generated incorrectly based on them. At that point, APIC-2 had already moved to the target version while APIC-3 was still waiting for its turn at the data conversion. The upgrade had to be aborted: APIC-3 and APIC-1 were returned to the original version, and APIC-2 was decommissioned, reimaged and commissioned back. This pre-upgrade validation script had been run before the upgrade and did not flag anything related to this.

Activity

  1. monrog2 commented on Nov 12, 2026

    @monrog2
    Collaborator

    This script can query APIC APIs and execute user-level commands on each APIC.

    Please provide the exact query/command evidence that supports that we can actually check for symptoms associated with this behavior when queried from the APIC via user-level access.

  2. mari9825 commented on Nov 16, 2026

    @mari9825
    Author

    Please confirm the logs attached. Sensitive information has been masked.

    If the APIC is running a CIMC version earlier than the version that includes the fix for CSCwe61589, it is important to verify the serial number using the acidiag show cimc command.

    acidiag show cimc

    ==================== CIMC DETAILS ====================
    PID : APIC-SERVER-L3
    Serial Number : UCSC-C220-M5 <<< 
    Product Name : UCSC-C220-M5
    

    The following logs were collected after the upgrade failed.

    moquery -c faultInst -f 'fault.Inst.highestSeverity=="critical"'
    Total Objects shown: 1
    
    
    fault.Inst
    
    code : F3031
    ack : no
    alert : no
    annotation :
    cause : cert-invalid
    changeSet :
    childAction :
    created : 2026-09-08T18:03:54.958+09:00
    delegated : no
    descr : Node Certificate is invalid: Failed to parse the subject line as a valid ACI fabric certificate AND Invalid Serial Number AND Invalid Product ID
    dn : uni/fabsslcomm/ifmcertnode-2/fault-F3031
    domain : security
    extMngdBy : undefined
    highestSeverity : critical
    lastTransition : 2026-09-08T18:05:59.218+09:00
    lc : raised
    modTs : never
    occur : 1
    origSeverity : critical
    prevSeverity : critical
    rn : fault-F3031
    rule : pki-fabric-nodesslcertificate-node-cert-invalid
    severity : critical
    status :
    subject : security-provider
    title :
    type : operational
    uid :
    userdom : all
    

    dbgexp_tsod-techsupport_sysid-2_2026-09-08T18-57JST_logs_3of3/data/log/svc_ifc_appliancedirector.bin.warnplus.log

    5057||2026-09-07T12:19:27.450857038+00:00||proc_app||ERROR||||Failed to find matching system-product-name||../common/src/ifcutils/ifc_pid_check.cc||104
    

    acidiag verify apic
    dbgexp_tsod-techsupport_sysid-2_2026-09-08T18-57JST_1of3/var/log/dme/oldlog/cli[LOG_ID]/verify_apic

    openssl_check: Fabric SSL certificate details
    subject=CN=[APIC_SERIAL_NUMBER],serialNumber=PID:UCSC-C220-M5 SN:[APIC_SERIAL_NUMBER]
    issuer=CN=Cisco APIC CA,O=Cisco System,ST=CA,C=US
    notBefore=Sep 7 12:18:11 2026 GMT <<< Generated at the time of the upgrade
    notAfter=Aug 30 12:18:11 2056 GMT
    openssl_check: passed
    subject=serialNumber = PID:UCSC-C220-M5 SN:[APIC_SERIAL_NUMBER], CN = [APIC_SERIAL_NUMBER]
    Certificate doesn't match APIC format
    apic_cert_format_check: failed
    
  3. tskanai1 commented on Nov 17, 2026

    @tskanai1

    @monrog2 @mari9825 I implemented the new check against CSCwe61589 and opened the PR #498.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions