Linux · critical · Storage, filesystems & NFS

mdadm array degraded, or missing after a reboot

A member has dropped out of the array, or the array did not assemble at boot because the configuration no longer matches the disks.

What you see

The array runs with one disk missing, or the device is absent at boot and the filesystem does not mount. /proc/mdstat shows the state.

What is actually wrong

A failing disk, a loose cable, or a mdadm.conf and initramfs that were not updated after the array was changed.

Codes and articles

mdadm degradedmd: super_writtenresyncremovedinactive arraymdadm: cannot open

Fixes (2)

Replace the failed member

Root shell1–8 hours depending on sizehigh risknot reversible

The array is degraded but running. Do this promptly — a second failure during a rebuild loses everything on a RAID5.

Read the state and identify the failed device.
Shell
```
cat /proc/mdstatsudo mdadm --detail /dev/md0
```
Check the health of every remaining member before rebuilding. A rebuild reads every sector of every disk and is exactly when a second marginal drive fails.
Shell
```
for d in /dev/sd{a,b,c,d}; do echo "== $d"; sudo smartctl -H $d; done
```
This is the step that separates a routine replacement from losing the array. If a second disk is showing pending sectors, back up before rebuilding, not after.

Remove the failed member.

Shell

sudo mdadm --manage /dev/md0 --fail /dev/sdc1 --remove /dev/sdc1

Partition the replacement to match, then add it.

Shell

sudo sfdisk -d /dev/sda | sudo sfdisk /dev/sdcsudo mdadm --manage /dev/md0 --add /dev/sdc1

Watch the rebuild. Do not reboot during it.
Shell
```
watch -n5 cat /proc/mdstat
```

Set up email alerting so the next failure is noticed on the day it happens.

Shell

sudo sed -i 's/^MAILADDR.*/MAILADDR admin@example.com/' /etc/mdadm/mdadm.confsudo mdadm --monitor --scan --test --oneshot

Confirm it workedmdstat shows all members [UU] with no rebuild in progress.

Shell

cat /proc/mdstat; sudo mdadm --detail /dev/md0 | grep -E 'State|Active|Working|Failed'

If you need to undo itNone — the replaced disk's data is rebuilt from parity.

Reassemble an array that did not come up

Root shell40 minuteshigh riskreversible

The array is inactive or missing after a reboot.

Look for the members and their superblocks.
Shell
```
sudo mdadm --examine /dev/sd[a-d]1 | grep -E 'Device Role|Array UUID|Events|State'
```
The Events counter is the key number. Members with matching counts can be assembled cleanly; a member several hundred behind was out of the array and must be re-added rather than assembled, or it will corrupt the data.

Assemble by UUID, which is stable across device name changes.

Shell

sudo mdadm --assemble --scansudo mdadm --assemble /dev/md0 --uuid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

If it will not assemble because a member is behind, force it only after understanding which member and by how much.
Shell
```
sudo mdadm --assemble --force /dev/md0 /dev/sda1 /dev/sdb1
```
Once running, write the configuration and rebuild the initramfs — this is the step whose absence causes the array to vanish at every boot.
Shell
```
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.confsudo update-initramfs -u
```
The array is assembled from the initramfs before the root filesystem is available, so a configuration written only to /etc is not read at the time it is needed.
Check fstab uses UUIDs rather than /dev/md0, which can change number.
Shell
```
sudo blkid /dev/md0grep md /etc/fstab
```

Confirm it workedThe array assembles automatically after a reboot and the filesystem mounts.

Shell

sudo rebootcat /proc/mdstat

If you need to undo itSuperblocks are unchanged by assembly. --force can commit a stale member, so examine first.

Related faults

Where this stops. This write-up was written and checked by hand. It says what each step changes, how to confirm it worked and how to reverse it, and anything destructive is flagged before you reach it. If it does not match what your machine is doing, search the Support Centre for the exact code or message — and when something needs a person, get in touch.