Server RAID systems are built for uptime, with hardware controllers, hot spare drives, and sometimes many dozens of disks in several arrays. When one fails past what the redundancy covers, the situation is stressful, and mistakes made in the first hour often decide the outcome.
Typical causes are a second drive failing during a rebuild, a controller failure or lost configuration, a power event, or someone pulling the wrong drive. A hot spare normally starts rebuilding on its own, which puts heavy load on the surviving drives. If those drives have bad sectors, the rebuild can fail partway and leave the array offline or inconsistent.
Recovery usually means imaging every member drive, then reconstructing the array in software from the images. That requires finding the RAID level, stripe size, parity layout, and disk order, which a controller stores in its own metadata. Different controller brands store these differently, and the volume above may hold a file system, a virtual machine datastore, or a database.
If an array goes down, stop. Don't force a drive back online, run a rebuild, or initialize the volume. Don't accept a controller's offer to import a foreign configuration unless you know what it will do. Label each drive with its slot, power down in an orderly way, and save the controller's logs and configuration if you can read them.
Advanced · enterprise · server · raid
RAID 5 Explained
Single parity and the danger of the rebuild
RAID 5 stripes data across three or more drives and adds parity, calculated from the data and rotated across all drives. It survives the loss of any one drive by calculating the missing data from the rest.
While degraded, the array has no margin. A rebuild reads every sector of every surviving drive, and if any of them has a bad sector or fails during the process, the rebuild can stop or leave data corrupted.
Before rebuilding a degraded RAID 5, make sure there's a current backup. If a second drive drops out, stop: don't force drives back online or recreate the array. Each member can be imaged and the array reconstructed virtually, including its parity rotation and drive order.
RAID 6 adds a second, independently calculated parity block to each stripe, so the array survives two drive failures. It's common in large arrays of big drives, where rebuilds take a long time and a second failure during a rebuild is a real risk.
The tradeoff is write performance and usable capacity. Recovery is similar to RAID 5, but with a more complex parity scheme that has to be identified correctly.
A RAID 6 with one failed drive still has protection; with two failed drives, it has none. Treat a double-degraded RAID 6 like a degraded RAID 5: back up first, and don't force anything.
RAID 10 builds mirrored pairs and stripes data across the pairs. It's fast, rebuilds quickly by copying from a mirror partner, and is popular for databases and virtual machines.
It can survive several drive failures as long as both drives of the same mirror pair don't fail. Lose both halves of one pair, and every file striped across that pair is damaged.
When a RAID 10 fails, which drives were partners matters. Label drives with their exact slots, and don't swap or rebuild anything until you know which pairs are still intact.
RAID 50 and RAID 60 stripe data across several RAID 5 or RAID 6 groups. They let large arrays rebuild faster and confine a single failure to one group.
Each group tolerates its own failures: one drive per RAID 5 group, two per RAID 6 group. Exceed that in any one group, and the whole array is affected.
Reconstruction requires identifying how drives were grouped, the stripe size at both levels, and the parity rotation, all from the drives themselves if the controller's configuration is lost. Keep a record of how your arrays were built.
A rebuild that fails partway usually means another drive hit unreadable sectors or dropped out under the load. The array may now show as failed, or as partially rebuilt, with the new drive holding a mix of rebuilt and empty areas.
This is a critical moment. Retrying the rebuild, forcing the failed drive online, or recreating the array can overwrite parity and data with wrong values.
Power the system down if you can do so safely. Save the controller logs and write down the order of events and which drive was replaced. Every drive, including the one removed earlier, can then be imaged and the array reconstructed as it was before the rebuild began.
A hot spare is an idle drive in the array that the controller uses automatically to rebuild when a member fails. It shortens the time an array runs without protection, but it can also start a rebuild nobody noticed, on drives that are already weak.
Every array should send alerts when it becomes degraded, and someone has to actually receive them and act. A familiar story is an array that ran degraded for months before a second drive failed.
When an alert arrives, check backups first, then replace the drive. If the array has more than one problem, pause before letting anything rebuild.
Hardware RAID uses a dedicated controller card with its own processor, cache, and metadata format. Software RAID, such as Linux mdadm, Windows Storage Spaces, or the RAID built into ZFS, runs in the operating system and keeps its metadata on the drives in a documented format.
Software RAID is easier to move to new hardware, since any compatible system can assemble it. Hardware RAID can be tied to a controller family, and moving drives to a different brand of controller rarely works without reconstruction.
Firmware RAID, often built into motherboards, sits in between and can be the hardest to move. Whatever you use, document the array type, level, stripe size, and drive order.
A storage area network presents blocks of storage, called LUNs, to servers over Fibre Channel or iSCSI. Each server formats its LUN with its own file system, such as NTFS, VMFS, or ext4, so the SAN itself doesn't know what files it holds.
Failures can happen at several layers: the drives, the array's RAID and pooling, thin-provisioning metadata, snapshots, or the file system the server put on top. Each layer has to be rebuilt in order.
During a SAN incident, preserve the array's configuration and logs, avoid re-creating LUNs or volumes, and stop servers from writing to the affected storage. Vendor support and a recovery lab may both be needed.
Advanced · san · iscsi · lun · block storage
VMware Datastores and VMFS
Virtual machine storage on shared volumes
VMware ESXi stores virtual machines on datastores, usually formatted with VMFS. Each VM is a folder of files: configuration, virtual disks (a small VMDK descriptor plus a large data file), and snapshot delta files that hold the changes made since each snapshot.
Common problems include a datastore that won't mount after a storage failure, deleted VMs, and broken snapshot chains, where a delta file is missing or a descriptor no longer points to the right parent.
Don't delete or consolidate snapshots on a VM that has errors, and don't reformat a datastore that won't mount. Recover the underlying storage first, then the VMFS volume, then the VM files.
Advanced · vmware · vmfs · vmdk · datastore
Hyper-V Virtual Disks and Checkpoints
VHDX files, AVHDX chains, and merges
Hyper-V stores virtual disks as VHDX files. When a checkpoint is created, new writes go to a differencing disk, an AVHDX file, that depends on its parent. A chain of checkpoints means a chain of files, each needed to see the latest state.
Deleting AVHDX files by hand, moving files without their parents, or interrupting a merge can break the chain. The VM may then refuse to start or show older data.
If a VM's disk chain breaks, copy every related file somewhere safe before changing anything. The chain can usually be repaired by reconnecting the files in order, and if one file is damaged, the data in the remaining layers can still be recovered.
Sudden power loss can hit a server in several ways: drives that fail on restart, RAID controllers that lose cached writes because the cache battery was dead, file systems and databases caught mid-write, and power supplies damaged by surges when power returns.
On restart, watch for RAID warnings, file system checks, and database consistency errors. If the array reports several failed drives at once, a power or controller event is more likely than simultaneous drive failures, and forcing drives back online carelessly can make things worse.
A UPS that shuts servers down cleanly, a healthy controller cache battery, and tested backups prevent most of these situations.
Intermediate · power outage · server · ups · raid cache
When the Backup Server Fails
Recovering the recovery system
Backup servers and appliances are storage systems too, and they fail: drives die, repositories get corrupted, ransomware targets them, and deduplicated stores can lose the index that ties stored chunks back to files.
When a backup server fails, there are two goals: protect any production data that's still running, and recover the backup repository itself. Backup formats are often proprietary and deduplicated, so recovering a repository means rebuilding the storage first, then the backup software's own structures.
Keep at least one backup copy on separate storage, ideally offline or immutable, and test restores from that copy, not just from the primary backup target.