I have had one of those days today…
One of the mount points on my server had gotten a little tight for space, so I decided to grow it before it became a problem. Naturally I use LVM so that’s not an issue.. well assuming you have free space on the LVM physical volume that is.. Thankfully this is a server running under virtualisation, so adding an additional physical volume was a simple task and the expansion went as you would expect, flawlessly.
Whilst I was at it, I thought I’d upgrade this box from Debian Squeeze to Wheezy too. Already running Wheezy elsewhere and an upgrade I have done plenty of times to trust (mostly 😉 ). That went well, but there were a few little grumbles around grub complaining about a missing PV that worried me. So I took a snapshot (including VMs memory) before running the magical restart. I am VERY glad I did!
Alas it seems there is a bug in GRUB 1.99 that from my Googling seems to have been around a while! It seems as though it isn’t too happy with an LVM configuration with multiple PVs. Reverting my non-booting box back to it’s running snapshot I was able to attempt ‘update-grub’ and be greeted with a long list of errors regards PV0 not being found.
Thankfully, I think I have found a work around! Although it’s not pretty and a bit time consuming. Essentially present a new PV to the box that is large enough to accommodate all existing PVs data (and probably a little extra growth room), then move your data to this new PV from all the original PVs with pvmove. This will take some time, running pvmove -v in another session/terminal you can at least see progress of the migration.
Once all the data is in your new PV, remove the old PVs from the active VG, then remove them from the system with pvremove. (They can still be connected and show as disks, just not PVs). With that done, run update-grub again, and fingers crossed you should have a bootable system once more.
I would certainly recommend you fire off a new snapshot including memory before rebooting, it’s an excellent safety net to get you back on the box.
Hopefully this will help someone else get out of this situation if they are unlucky to find themselves in it.