This might be the 1000th blog posting on this general topic, but for some reason, the complexity of booting grows year over year, sort of like the tax code.
Back in 2009, Win and I built three low power servers, using Intel D945GCLF2 mini-ITX motherboards with Atom 330 processors. We put mirrored 1.5 Terabyte drives in them, and 2 GB of ram, and they have performed very well as pretty low power home servers. We ran the then-current Ubuntu, and only sporadically ran apt-get update and apt-get upgrade.
Fast forward to this summer. We wanted to upgrade the OS’s, but they had gotten so far behind that apt-get update wouldn’t work. It was clearly necessary to reinstall. Now one of these machines is our compound mail server, and another runs mythtv and various other services. The third one was pretty idle, just hosting about a terabyte of SiCortex archives. In a previous blog post I wrote about the month elapsed time it took me to back up that machine.
This post is about the adventure of installing Ubuntu 12.04 LTS on it. (LTS is long term support, so that in principle, we will not have to do this again until 2017. I hope so!)
Previously, SMART tools were telling us that the 2009 era desktop 1.5T drives were going bad, so I bought a couple of 3T WD Red NAS drives, like the ones in our Drobo 5N. Alex (my 14 year old) and I took apart the machine and replaced the drives, with no problem.
I followed directions from the web on how to download an ISO and burn it to a USB drive using MacOS tools. This is pretty straightforward, but not obvious. First you have to convert the iso to a dmg, then use dd to copy it to the raw device:
hdiutil convert -format UDRW -o ubuntu-12.04.3-server-amd64.img ubuntu-12.04.3-server-amd64.iso # Use diskutil list, then plug in a blank USB key >the image size, run diskutil list again to find the drive device. (In my case /dev/disk2) sudo dd if=ubuntu-12.04.3-server-amd64.img.dmg of=/dev/disk2 bs=1m # notice the .dmg extension that MacOS insists on adding diskutil eject /dev/disk2 (or whatever)
Now in my basement, the two servers I have are plugged into a USB/VGA monitor and keyboard switch, and it is fairly slow to react when the video signal comes and goes. In fact it is so slow that you miss the opportunity to type “F2” to enter the BIOS to set the boot order. So I had to plug in the monitor and keyboard directly, in order to enable USB booting. At least it HAS USB booting, because these machines do not have optical drives, since they have only two SATA ports.
Anyway, I was able to boot the Ubuntu installer. Now even at this late date, it is not really well supported to install onto a software RAID environment. It works, but you have to read web pages full of advice, and run the partitioner in manual mode.
May I take a moment to rant? PLEASE DATE YOUR WEB PAGES. It is astonishing how many sources of potentially valuable information fail to mention the date or versions of software they apply to.
I found various pieces of advice, plus my recollection of how I did this in 2009, and configured root, swap, and /data as software RAID 1 (mirrored disks). Ubuntu ran the installer, and… would not reboot. “No bootable drives found”.
During the install, there was an anomaly, in that attempts to set the “bootable” flag on the root filesystem partitions failed, and when I tried it using parted running in rescue mode, it would set the bootable flag, but clear the “physical volume for RAID” flag.
I tried 12.04. I tried 13.04. I tried 13.04 in single drive (no RAID). These did not work. The single drive attempt taught me that the problem wasn’t the RAID configuration at all.
During this process, I began to learn about GPT, or guid partition tables.
Disks larger than 2T can’t work with MBR (master boot record) style partition tables, because their integers are too small. Instead, there is a new GPT (guid partition table) scheme, that uses 64 bit numbers.
Modern computers also have something called UEFI instead of BIOS, and UEFI knows about GPT partition tables.
The Ubuntu installer knows that large disks must use GPT, and does so
Grub2 knows this is a problem, and requires the existence of a small partition flagged bios_grub, as a place to stash its code, since GPT does not have the blank space after the sector 0 boot code that exists in the MBR world (which grub uses to stash code).
So Ubuntu creates the GPT, the automatic partitioning step creates the correct mini-partition for grub to use, and it seems to realize that grub should be installed on both drives when using an MD filesystem for root. (it used the command line grub-install /dev/sda /dev/sdb) Evidently the grub install puts a first stage loader in sector 0, and the second stage loader in the bios_grub partition.
Many web pages say you have to set the “bootable” flag on the MD root, but parted will not let you do this,because in GPT, setting a “bootable” flag is forbidden by the spec. Not clear it would work anyway because when you set it, the “physical volume for raid” flag is turned off.
The 2009 Atom motherboards do not have a UEFI compatible BIOS, and are expecting an MBR. When they don’t find one, they give up. If they would just load the code in sector 0 and jump to it it would work. I considered doing a bios update, but it wasn’t clear the 2010 release is different in this respect.
So the trick is to use FDISK to <create an MBR> with a null partition. This is just enough to get past the Atom BIOS’ squeamishness and have it execute the grub loader, which then works fine using the GPT. I got this final trick from http://mjg59.dreamwidth.org/8035.html whose final text is
boot off a live CD and run fdisk against the boot disk. It’ll give a bunch of scary warnings. Ignore them. Hit “a”, then “1”, then “w” to write it to disk. Things ought to work then.
The sequence of steps that worked is:
Run the installer Choose manual disk partitioning Choose "automatically partition" /dev/sda This will create a 1 MB bios_grub partition and a 2GB swap, and make the rest rootDelete the root partition Create a 100 GB partition from the beginning of the free space Mark it "physical volume for RAID" with a comment that it is for root Use the rest of the free space (2.9T) to make a partition, mark it physical volume for raid. Comment that it is for /data Change the type of the swap partition to "physical volume for raid" Repeat the above steps for /dev/sdb Run "configure software RAID" Create MD volume, using RAID 1 (mirrored) Select 2 drives, with 0 spares Choose the two swap partitions Mark the resulting MD partition as swap Create MD volume, RAID 1, 2, and 0 Select the two 100 GB partitions Mark them for use as EXT4, to be mounted on / Create MD volume, RAID 1, 2, and 0 Select the two 2.9T partitions Mark them for use as EXT4, to be mounted on /data (I considered BTRFS, but the most recent comments I could find still seem to regard it as flakey) Save and finish installing Ubuntu Pretend to be surprised when it won't boot. "No bootable disks found" Reboot from the installer USB, choose Rescue Mode Step through it. Do not mount any file systems, ask for a shell in the installer environment. When you get a prompt, fdisk /dev/sda a 1 w Then fdisk /dev/sdb a 1 w ^d and reboot. Done Now I have a working Ubuntu 12.04 server with mirrored 3T drives.