My home server is a Dell PowerEdge 2950 with 4 SAS disks, 1 for OS, 2 in active RAID1, and a hot spare.
I’ve been getting regular emails (now that smartctl can send emails with postfix fixed and hostname in all forms is correct) telling me a drive is about to fail.
Need to pay attention to the integrity of those backup files! And copy them elsewhere, delete them, occasionally restart mysql when it ran out of space. And delete /var/lib/mysql/tc.log – a zero-byte file that prevented restarting of mysql.service.
It’s getting to be a pain. But, chance to test RAID mirror or upgrade Ubuntu.
And then,…
I looked closer at the output of the df command df -h | grep -E '^/dev' | grep -Ev "loop|tmpfs"
And saw nothing like this:
/dev/sdc1 133G 83G 44G 66% / /dev/mapper/vg_data-var_raid 133G 67G 60G 54% /var
No “mapper” – no LVM on RAID1. WTH?!? I’d created a physical volume:
pvs
PV VG Fmt Attr PSize PFree
/dev/md127 vg_data lvm2 a-- <136.00g 0
And a volume group:
vgs
VG #PV #LV #SN Attr VSize VFree
vg_data 1 1 0 wz--n- <136.00g 0
Yet somehow didn’t make a logical volume and mount my “/var” onto that.
/embarrassing admission
So, a Logical Volume was created and mounted to a temporary location, rsync was run to get the contents of /var onto the RAID, when everything seemed good, I updated /etc/fstab, then, finally, pulled the trigger and mounted the RAID onto /var…
And that’s how I got this here hole in my other foot.
Me, realizing I’d forgotten something
All the data was still on the original /var, along with / and /home, and so backups (which went into /home/$USER/...) filled the drive.
While strategizing how to shut down as many services as possible, umount --force the RAID, mv -v /var /var-ORIG, mkdir -v /var, mount -a and the possible repercussions of that (have shot both feet already, what’s next?!?), I attended a meeting of Bay Area LUG (San Fransisco) and asked Michael Paoli for advice. Wise choice!
His simple and elegant solution has me kicking myself for not thinking of it on my own:
mkdir -v /tmp/root
mount /dev/sdc1 /tmp/root
rm -fr /tmp/root/var/{largeFiles,moreLargeFiles,foldersOfLargeFiles}
And, the underlying root file system is no longer filling up. Just had to remount the root file system to a temporary location and I could access it without interfering with the RAID-mounted /var. Brilliant!
But wait, there’s more to this story…
