IBM hardware (especially IBM Z and IBM Power systems) distinguishes and logs transient vs permanent faults using a layered error detection, classification, and telemetry logging pipeline built into firmware, service processors, and hardware error correction units.

The goal is:

Capture every fault event, classify whether it is temporary or persistent, and decide whether to retry, recover, or permanently deconfigure the component.

1. Core idea

A fault is not just “an error”—IBM systems classify it based on repeatability, recoverability, and hardware evidence over time.

So:

• Transient fault = temporary glitch
• Permanent fault = structural hardware failure

2. Where fault logging happens

Faults are logged at multiple layers:

A. Hardware level

• CPU execution units
• cache controllers
• memory controllers
• I/O adapters

B. Firmware level (LIC / PHYP / PowerVM)

• error classification engine
• recovery decision logic
• persistence tracking

C. Service processor (SP / FSP / HMC)

• long-term fault logs
• predictive analytics storage
• maintenance reporting

3. Types of faults captured

A. Transient faults

Examples:

• cosmic ray bit flips
• temporary signal noise
• timing glitches
• single ECC-corrected memory error

👉 Usually recoverable

B. Permanent faults

Examples:

• repeated ECC failures at same address
• stuck CPU pipeline stage
• failing memory chip
• defective I/O adapter hardware

👉 Requires deconfiguration or replacement

4. How IBM distinguishes transient vs permanent faults

Step 1: Error detection

Hardware detects:

• ECC correction
• parity error
• instruction retry
• I/O timeout

Step 2: Immediate classification (fast path)

Firmware assigns initial tag:

• “corrected transient”
• “uncertain”
• “fatal / uncorrectable”

Step 3: Historical correlation (critical step)

System checks:

• Has this error occurred before at same location?
• Is error rate increasing?
• Is pattern recurring across time?

👉 This is key to classification.

Step 4: Error threshold analysis

IBM systems use thresholds such as:

• number of corrected errors per time window
• repeated errors per memory address
• retry count on CPU or I/O operation

If thresholds exceeded:

• transient → permanent suspicion

Step 5: Stability observation period

System may:

• continue monitoring component
• track error recurrence rate
• evaluate performance degradation trends

Step 6: Final classification decision

Transient fault if:

• isolated event
• no recurrence
• successfully corrected (ECC retry, instruction retry)
• no performance degradation

Permanent fault if:

• repeated identical errors
• error frequency increases
• hardware consistency check fails
• retry attempts fail

5. How faults are logged

A. Hardware error registers

• immediate capture of fault type
• CPU/memory/I/O error codes

B. Firmware logs

Firmware stores:

• error timestamps
• component IDs
• recovery actions taken
• severity classification

C. Service processor logs

Persistent logging includes:

• historical fault trends
• predictive failure indicators
• field replaceable unit (FRU) mapping

D. Operating system logs

OS receives:

• abstracted error events (not raw hardware noise)
• only significant or persistent faults

6. Key mechanisms used for classification

A. ECC correction tracking

• single-bit corrected → likely transient
• repeated corrections → suspect permanent issue

B. Retry counters

• instruction retry success → transient
• repeated failure → permanent

C. Error clustering

• multiple errors in same memory region → hardware degradation

D. Time-based correlation

• short bursts → transient noise
• long-term pattern → structural failure

7. Actions after classification

If transient:

• log event only
• continue operation
• no hardware removal

If permanent:

• isolate component
• deconfigure CPU/memory/I/O path
• trigger replacement alert
• reroute workloads

8. Example flow (memory error)

Case 1: transient

• ECC corrects single bit flip
• no recurrence
• logged as “corrected error”
  👉 system continues normally

Case 2: permanent

• repeated ECC errors at same location
• chipkill engaged frequently
• degradation trend detected
  👉 memory page or DIMM is retired

9. Why IBM is good at this classification

Because it uses:

• continuous telemetry (every error is recorded)
• hardware-level error detection (very granular)
• long-term historical baselines
• firmware-level decision intelligence

10. Simple mental model

Think of IBM fault logging as:

A continuous diagnostic system where every hardware error is recorded, analyzed for recurrence patterns, and classified over time as either a harmless transient glitch or a persistent hardware defect requiring isolation.

11. Key takeaway

IBM hardware distinguishes transient vs permanent faults by:

• capturing all errors at hardware level
• tracking recurrence and error frequency
• analyzing correction success (ECC/retry behavior)
• correlating long-term patterns via firmware and service processor logs
• isolating components only when persistence indicates real failure

👉 Result: most transient faults are invisible to applications, while permanent faults are proactively contained before system impact