As a few of you probably noticed, pipeline1, also known as seed.sdsu.edu, has been down at least since wednesday or thursday of last week. One of the drives was having “problems”, we’ll say, that were forcing pipe1 to be unable to boot. The actual boot drive wasn’t the issue. The second drive was the issue. It was formatted with an lvm2 filesystem. Also known as “logical partitions”. I guess it’s one way to get multiple partitions onto one drive. The catch is that it makes it really difficult to fsck it if there’s problems. After spending quite a while on it, here’s what I’ve done to get it fixed.

1: Boot off of an ubuntu install disc. It gives me graphical windows, a terminal, and easy root access. Tools used: Applications->Accessories->Terminal

2: I had to install lvdisplay, part of the lvm2 package, to get a look at our logical partitions, but I’ll tell you what they are so you don’t have to do that. Our two logical partitions are /dev/VolGroup00/LogVol00 and /dev/VolGroup00/LogVol01.

3: Here is the SECRET TRICK. If you try to fsck either of those, it’ll give you an error. First it will tell you that no such file exists, and second it will tell you that it tried anyways, and there’s a bad super block. The second statement is misleading, concentrate on the first. So what I did was go to /dev/VolGroup00, and take a look at the two partitions. Both of them were symlinks! *gasp*

4: Those links pointed to the REAL partitions. /dev/mapper/VolGroup00-LogVol00 and /dev/mapper/VolGroup00-LogVol01.

5: I ran fsck on the problem volume (volume 00) and lo and behold, success! All the orphaned inodes and corrupt linked lists are finding their parents and…being trimmed? I don’t know. Suffice to say fsck is currently running and hopefully pipe1 will be back up soon.