Add Storage TSG: repair jobs restart and never complete on a failing high-latency disk - #327
Open
John Neemes (1008covingtonlane) wants to merge 9 commits into
Conversation
New troubleshooting guide for S2D storage repair/regeneration jobs that reset and never complete when a single disk is failing with high latency and IO errors. Covers detection surfaces (PowerShell, event logs, cluster log, Azure portal, FCM/WAC), the free-space and reserve check before retiring, the retire-and-rebuild resolution, pre-retire gotchas, and prevention. Adds the guide to the Storage index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Copilot started reviewing on behalf of
John Neemes (1008covingtonlane)
July 10, 2026 23:32
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Storage TSG for diagnosing and resolving repair jobs blocked by a failing high-latency disk.
Changes:
- Documents symptoms, diagnostics, capacity checks, and disk replacement.
- Adds the guide to the Storage index.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md |
Adds the troubleshooting procedure. |
TSG/Storage/README.md |
Links the new guide. |
Resolves the four Copilot PR-review findings: (1) the pre-retire one-fault-domain check now also filters OperationalStatus (Abnormal Latency / Lost Communication) and surfaces the node, so a second stale-Healthy failing disk is not missed; (2) the Step 3 free-space query excludes the retiring disk, shows per-node free capacity, and no longer truncates to five rows; (3) the worked example now distinguishes sufficient evacuation headroom from full reserve (not restored until the replacement drive is added); (4) Step 6 uses Remove-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -UniqueId ...) because -UniqueId is not a valid parameter. Also adds usability sections from a persona review: a TL;DR fastest-path, an Impact and ownership summary, a Glossary, a Scope note distinguishing RDMA/network issues, an explicit workload-impact note, a When to escalate section, and OEM/hardware framing for the physical swap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Collaborator
Author
|
Thanks for the review. Addressed all four findings in 9d63f9c:
|
Replace the "TL;DR" callout with an "## Overview" section that leads with plain-language what, why, and impact and keeps the fast at-a-glance path. Correct the resiliency description: name the Azure Local default (three-way mirror on three or more nodes, two-way mirror on two-node) and cover the other Storage Spaces Direct schemes (parity, dual parity, nested resiliency, and mirror-accelerated parity), plus Simple with no redundancy. Explain that a slow, high-latency but SMART-healthy disk is not auto-failed under any scheme, and that retire then rebuild restores redundancy on any resilient volume while a Simple volume must be restored from backup. Confirm ResiliencySettingName before acting. Update the Glossary and a Symptoms bullet to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Reviewer consistency nit: Step 6 named the sibling by description while Related Issues uses exact paths. Point Step 6 at TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md the same way. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd
Sanding pass from the tsg-forge grader: - Multi-surface discoverability 6/8 -> 8/8: add Windows Admin Center in the Azure portal (the same cluster Drives view for an Arc-connected cluster) and a Storage diagnostic logs (on disk) note (Get-SDDCDiagnosticInfo / Support Diagnostics Tool collect the storage subsystem's own on-disk logs). The section now shows the failing drive across all eight admin surfaces. - Prose: replace em-dashes in the resiliency-type bullets with colons (house style). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d7f2348-e3be-4fe4-8598-6bf5dcb4050f
…hmetic 1. The volume-health snippet selected OperationalDetails, which is not an MSFT_VirtualDisk property (per the class MOF it has OtherOperationalStatus- Description and DetachedReason, not OperationalDetails), so that column always rendered blank. Use OtherOperationalStatusDescription. 2. The Step 3 reserve worked example subtracted the retiring drive's full 2.4 TB size from pool-free, but only ~0.4 TB of the free space sits on that drive (2.4 TB total, ~2.0 TB used). Reworded so the numbers are consistent: surviving free before the rebuild is ~10.4 TB, and ~8.4 TB is the free that remains after relocating the ~2.0 TB. The conclusion (fits, restores redundancy, but below the 9.6 TB full-reserve target) is unchanged.
Collaborator
|
Read the full guide and checked every command, parameter, property, health-fault string, and event ID in it against the product PowerShell reference and the emitted health-fault catalog. The six fault strings all match the catalog. The remediation cmdlets and their parameters line up, including the detail that Remove-PhysicalDisk has no -UniqueId and has to take -PhysicalDisks with -StoragePoolFriendlyName, which is correct. All of the PowerShell blocks parse clean, the reserve-capacity worked example is arithmetically consistent, and the per-resiliency-scheme guidance (mirror, parity, nested, simple) is right. I could not exercise the retire and rebuild on hardware this time, and a failing-disk fault cannot be injected on a healthy cluster in any case, so that path is validated against the cmdlet reference and the source rather than a live run. Two things to fix: Step 6 calls Enable-PhysicalDiskIndication, which is not a real cmdlet and will fail at runtime with a "term is not recognized" error. The cmdlet is Enable-PhysicalDiskIdentification (only the Identification spelling exists in the Storage module, Windows Server 2016 through 2025; the Indication spelling has no reference page). Change it to Step 2, the "Identify the failing physical disk" one-liner filters OperationalStatus on 'Abnormal Latency' only, but the guide itself says the drive can present as "Abnormal Latency" or "Lost Communication". A drive in the Lost Communication state returns nothing from that command. The Get-HealthFault line in the same step and the pre-checks filter (OperationalStatus -ne 'OK') both still surface it, so it is not fatal, but the labeled identify command should catch both: |
…sk filter to Lost Communication - Step 6 LED-locate used Enable-PhysicalDiskIndication, which does not exist (Learn 404); the real Storage-module cmdlet is Enable-PhysicalDiskIdentification (Learn 200, WindowsServer2025-ps). As written the command throws 'not recognized as a cmdlet' on a node. - Step 2 identify one-liner matched only 'Abnormal Latency', missing the 'Lost Communication' operational state this guide itself names; broadened the -match to cover both.
…ce caveats Applied from a tsg-forge grading run (live L4 loop re-proven on solution train 12.2610, plus a multi-model persona panel across two frontier model families). Safety (both safety lenses, both models): - Step 4 now opens with a four-point WARNING gate repeating the preconditions at the point of the destructive command, and the last-copy caveat now precedes the command rather than following it. - Added a mandatory read-back of the UniqueId before the retire, with an explicit escalation path when it does not match. - Documented that a retire is reversible. Correctness (found by the live run and by review): - Set-PhysicalDisk -Usage takes the enum AutoSelect, not the hyphenated Auto-Select that Get-PhysicalDisk displays. The hyphenated form fails outright. - The HighLatency fault multiple said "thousands to millions"; the guide's own worked example is about 585 times. - Reliability counters can return nulls or zeros behind some HBAs and NVMe, so empty counters are not evidence of a healthy drive. - Lost-communication and flapping events can be a cable, expander, or backplane fault, so the shared storage path is now ruled out before condemning a drive. - The 0x133 attribution now requires dump-stack and bugcheck-parameter confirmation instead of being asserted. Accessibility: - Added an acronym table (S2D, CSV, CAU, SDDC, SMART, HBA, DPC, RDMA) and an explicit note that angle-bracket tokens are placeholders.
…ep 6 gate) A third frontier model family (xAI) was added to the review panel after the previous third slot was found to be returning empty. It caught defects the first two models approved. Correctness: - "max latency in seconds, not milliseconds" contradicted the worked example in the same paragraph, which cites ReadLatencyMax = 11,708 ms. - The guide framed the failing drive as "still Healthy". That holds for the high-latency variant, but the lost-communication variant reports HealthStatus Warning, which a live lab run on this build confirmed. Filtering on HealthStatus alone misses it, so the symptom list now documents both variants and directs filtering on OperationalStatus. - "Get-StorageJob returns nothing" is not a valid rebuild-complete predicate, because it also returns Optimize, Rebalance, and Trim jobs and can block Step 6 indefinitely. Replaced with an explicit Repair/Regeneration predicate. Safety: - Step 4's read-back told the reader to confirm the node but printed no node. It now reads Get-StorageNode -PhysicallyConnected, and also surfaces MediaType. - Step 6 runs Remove-PhysicalDisk, which is irreversible and often executed in a new session hours later, but had no identity re-check. It now re-reads the drive, hard-stops unless it is still Retired and evacuated, separates lighting the bay from removing the disk, and requires matching the chassis serial sticker before anything is unplugged. - The reversibility note was unbounded. It is now scoped to undoing a wrong retire, with an explicit warning against un-retiring a genuinely failing drive mid-rebuild.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new Storage troubleshooting guide,
TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md, and links it from the Storage index.Why
There is no existing TSG for the common Storage Spaces Direct failure mode where
*-Repair/*-Regenerationstorage jobs reset toward 0% and never complete, with a volume stuck atOperationalStatus = No Redundancy. The usual cause is a single physical disk failing with very high latency and a high IO error rate while still reportingHealthStatus = Healthy(OperationalStatus = "OK, Abnormal Latency"). The same disk can also trip aDPC_WATCHDOG_VIOLATION(bugcheck 0x133) on its host node under write load, so a node crash and the stuck repair often share one cause. The condition is frequently misread as a pool-capacity problem.What the guide covers
PhysicalDisk.HighLatency.Outlier.AverageIO,PhysicalDisk.HighErrorCount.Outlier.AverageIO,VirtualDisks.NoRedundancy,VirtualDisks.LastCopy).disk153; Storage Spaces 203/205/207/209/302/304/305/312), cluster log, Azure portal, Failover Cluster Manager and Windows Admin Center.Set-PhysicalDisk -Usage Retired, monitor the rebuild jobs, thenRemove-PhysicalDisk, with risk labels and a last-copy caveat.All commands, fault strings, and event IDs are from an observed case and are scrubbed of environment-specific identifiers.