RAID1 farce

My home server is a Dell PowerEdge 2950 with 4 SAS disks, 1 for OS, 2 in active RAID1, and a hot spare.

I’ve been getting regular emails (now that smartctl can send emails with postfix fixed and hostname in all forms is correct) telling me a drive is about to fail.

Need to pay attention to the integrity of those backup files! And copy them elsewhere, delete them, occasionally restart mysql when it ran out of space. And delete /var/lib/mysql/tc.log – a zero-byte file that prevented restarting of mysql.service.

It’s getting to be a pain. But, chance to test RAID mirror or upgrade Ubuntu.

And then,…

I looked closer at the output of the df command df -h | grep -E '^/dev' | grep -Ev "loop|tmpfs"

And saw nothing like this:

/dev/sdc1                     133G   83G   44G  66% /
/dev/mapper/vg_data-var_raid  133G   67G   60G  54% /var

No “mapper” – no LVM on RAID1. WTH?!? I’d created a physical volume:

 PV         VG      Fmt  Attr PSize    PFree
 /dev/md127 vg_data lvm2 a--  <136.00g    0

And a volume group:

 VG      #PV #LV #SN Attr   VSize    VFree
 vg_data   1   1   0 wz--n- <136.00g    0

Yet somehow didn’t make a logical volume and mount my “/var” onto that.

/embarrassing admission

So, a Logical Volume was created and mounted to a temporary location, rsync was run to get the contents of /var onto the RAID, when everything seemed good, I updated /etc/fstab, then, finally, pulled the trigger and mounted the RAID onto /var

And that’s how I got this here hole in my other foot.

Me, realizing I’d forgotten something

All the data was still on the original /var, along with / and /home, and so backups (which went into /home/$USER/...) filled the drive.

While strategizing how to shut down as many services as possible, umount --force the RAID, mv -v /var /var-ORIG, mkdir -v /var, mount -a and the possible repercussions of that (have shot both feet already, what’s next?!?), I attended a meeting of Bay Area LUG (San Fransisco) and asked Michael Paoli for advice. Wise choice!

His simple and elegant solution has me kicking myself for not thinking of it on my own:

mkdir -v /tmp/root
mount /dev/sdc1 /tmp/root
rm -fr /tmp/root/var/{largeFiles,moreLargeFiles,foldersOfLargeFiles}

And, the underlying root file system is no longer filling up. Just had to remount the root file system to a temporary location and I could access it without interfering with the RAID-mounted /var. Brilliant!

But wait, there’s more to this story…

By admin

Linux admin, JS / React / Python / Perl developer.