HOWTO Gentoo Install on Software RAID mirror and LVM2 on top of RAID

From Gentoo Linux Wiki

[edit] Intro

This howto describes little additions to normal Gentoo x86 handbook installation documentation in order to setup Gentoo for server systems with software RAID mirror (RAID1) and LVM2 on top of RAID1 for easy partition managment. This HowTo deals with two IDE harddisks participating in RAID1 and they are located as /dev/hda, /dev/hdg. Your harddisks setup may differ of course.

Example server I used for this HowTo was 1U with Intel S845WD-E motherboard which has Promise Fasttrak RAID controller integrated. As with 2.6 kernels there is no ataraid, pdcraid in main kernel tree anymore, it's better to go ahead with standard softraid for 2.6 kernel. As this server also had cdrom and I wanted it to be also available as boot device and as it's not possible to boot harddisks behind Promise IDE controller I had to choose the following layout - HDD1 /dev/hda, CDROM /dev/hdb, HDD2 /dev/hdg (secondary Promise IDE interface).

[edit] Initial setup

• Boot LiveCD as usual, and load softraid module for raid1:

modprobe raid1

• Partition your harddisks and set partition types to Linux raid autodetect

[edit] There are two ways of partitioning

A) using "fdisk" B) using "cfdisk"

A) first we'll use fdisk command to perform partitions.

fdisk /dev/hda
fdisk /dev/hdg

I chose to have a following partition setup:

primary part1 -> boot 100MB
primary part2 -> swap 512MB
primary part3 -> root 1GB
primary part4 -> LVM2 for /usr /var /opt /tmp /home XGB

Alternate config:

primary part1 -> boot 100MB
primary part2 -> LVM2 for swap, / and whatever other filesystems like /var /home as you wish.

To set partition type to Linux raid autodetect, in fdisk press t and then type fd as the partition hex code.

# so the partition schema looks like that:
Device Boot         Start         End      Blocks   Id  System
/dev/hda1               1          13      104391   fd  Linux raid autodetect
/dev/hda2              14          76      506047+  fd  Linux raid autodetect
/dev/hda3              77         259     1469947+  fd  Linux raid autodetect
/dev/hda4             260       19457   154207935   fd  Linux raid autodetect

Device Boot         Start         End      Blocks   Id  System
/dev/hdg1               1          13      104391   fd  Linux raid autodetect
/dev/hdg2              14          76      506047+  fd  Linux raid autodetect
/dev/hdg3              77         259     1469947+  fd  Linux raid autodetect
/dev/hdg4             260       19457   154207935   fd  Linux raid autodetect

• Do not forget to make both partion 1 bootable otherwise you will cry at the reboot stage!
• We can setup the first disk partitions and then copy the entire partition table to the second disk with the following command:

sfdisk -d /dev/hda | sfdisk /dev/hdb

• If you get a message sfdisk doesn't like your partitions because a partition does not end on a cylinder boundary, you can override that.

Having to align the partition boundaries with the cylinders was a DOS legacy issue, and was not something that would cause a problem for Linux. You can override this by giving the --Linux option, that stands for 'do not complain about things irrelevant for Linux'

sfdisk -d /dev/hda | sfdisk -L /dev/hdb

B) By using "cfdisk", it is more easy way to partition over physical drives.

cfdisk /dev/hda
cfdisk /dev/hdg

there are many options available, select from menu "type" or by pressing "t" and set the partition type to "md" as it is for raid partition. Don't forget to make "boot" your boot partition from the option menu. Both drives will have same partition table of same size.

• Creating raid arrays for partition mirroring:

Linux has to have at least 1 non-raid partition to boot from, this can be a RAID1 array since RAID1 makes an exact copy of the disk. With RAID1 Disk1 and Disk2 will be exactly the same and therefore the disks will be separately mountable like normal. The reason why linux can't be booted from software Raid0 arrays is simple, the kernel has the drivers that are needed to read the raid array so the boot loader will not be able to access the kernel. It's the chicken or the egg problem. A RAID0 array or any RAID array that utilizes striping cannot be mounted without the raid software since all data spans multiple drives, without the software it's unreadable. This is not an issue with RAID1 (Mirrored) arrays since the disks themselves contain all the information for every partition and can be addressed individually just like any single partitioned disk. Because of this feature and to make sure in the event of a failure we can get back up and running quickly we will create a RAID1 array for /boot to allow us to have a backup copy of our kernels and other important files.

mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/hda1 /dev/hdg1
mdadm --create --verbose /dev/md2 --level=1 --raid-devices=2 /dev/hda2 /dev/hdg2
mdadm --create --verbose /dev/md3 --level=1 --raid-devices=2 /dev/hda3 /dev/hdg3
mdadm --create --verbose /dev/md4 --level=1 --raid-devices=2 /dev/hda4 /dev/hdg4

mknod /dev/md1 b 9 1
mknod /dev/md2 b 9 2
mknod /dev/md3 b 9 3
mknod /dev/md4 b 9 4

If you intend to create your software raid mirror using only one drive the below command will be helpful. The missing keyword allows you to create an array with only one drive. If you create your array with --raid-devices=1, you will be able to add another drive, but it will only be a spare. This of course is useless in a mirror.

mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/hda1 missing

• NB! Wait for arrays initialization being done:

# execute "watch -n1 'cat /proc/mdstat'" to see its status
livecd root # watch -n1 'cat /proc/mdstat'
Personalities : [raid1]
md4 : active raid1 hdg4[1] hda4[0]
     154207808 blocks [2/2] [UU]
     [>....................]  resync =  4.0% (6240576/154207808) finish=45.5min speed=54148K/sec
md3 : active raid1 hdg3[1] hda3[0]
     1469824 blocks [2/2] [UU]
md2 : active raid1 hdg2[1] hda2[0]
     505920 blocks [2/2] [UU]
md1 : active raid1 hdg1[1] hda1[0]
     104320 blocks [2/2] [UU]
unused devices: <none>

• Create mdadm.conf file for RAID setup reference:

# paste this inside
DEVICE          /dev/hda*
DEVICE          /dev/hdg*
ARRAY           /dev/md1 devices=/dev/hda1,/dev/hdg1
ARRAY           /dev/md2 devices=/dev/hda2,/dev/hdg2
ARRAY           /dev/md3 devices=/dev/hda3,/dev/hdg3
ARRAY           /dev/md4 devices=/dev/hda4,/dev/hdg4
MAILADDR        root@localhost

OR

mdadm --detail --scan > /etc/mdadm.conf

If you specify a MAILADDR the Gentoo init scripts will start mdmadm in monitor mode, and an email will be sent when any raid failures are detected.

[edit] Filesystems setup

• Create boot partition filesystem

mke2fs -j /dev/md1

• Create swap

mkswap /dev/md2
swapon /dev/md2

• Create root filesystem
  • I chose ReiserFS for 2 reasons:
    1. seems to perform faster than ext3
    2. stable online resizing feature which plays nicely with lvm

mkreiserfs /dev/md3

• Next we need to load LVM2 module:

modprobe dm-mod

• We need to put a device filter into /etc/lvm/lvm.conf to prevent errors in dealing with the RAID devices.

Also, we don't want to scan media like CD drives and such...

mkdir -p /etc/lvm
echo 'filter=["a|dev/md4|", "r/.*/"]' >/etc/lvm/lvm.conf

• Scan for LVM partition

vgscan
# it doesn't find anything...that's ok

• Creating LVM physical volume on md3

pvcreate /dev/md4

• Verify that the PV was created properly

pvdisplay

If the PV Name listed is not /dev/md4 but is instead one of /dev/hda4 or /dev/hdg4, you probably mis-typed the devices filter above.

NOTE: This didn't work for me, "pvdisplay /dev/md4" worked. --DR 06-09-21

• Creating VG volume group

vgcreate vg /dev/md4

• Verify that the VG was created successfully:

vgdisplay vg

Again, if the device filter line above is not correct, you'll get "Found Duplicate PV" errors, and the VG will not be created

• Creating needed partitions on LVM:

# Ignore errors "/etc/lvm/backup: fsync failed: Invalid argument" from now on
# You can extend partition later or make bigger initially - as you wish
lvcreate -L10G -nusr  vg
lvcreate -L5G  -nhome vg
lvcreate -L5G  -nopt  vg
lvcreate -L10G -nvar  vg
lvcreate -L2G  -ntmp  vg

• Creating filesystems on top of partitions

mkreiserfs /dev/vg/usr
mkreiserfs /dev/vg/home
mkreiserfs /dev/vg/opt
mkreiserfs /dev/vg/var
mkreiserfs /dev/vg/tmp

• • You could also use the Bash-tastic method ;)

for i in usr home opt var tmp; do mkreiserfs /dev/vg/$i; done;

[edit] Mounting

Creating needed mount points and mounting the filesystems:

mount /dev/md3 /mnt/gentoo/
cd /mnt/gentoo
mkdir boot usr home opt var tmp
chmod 1777 /mnt/gentoo/tmp
#if you have /var/tmp on a seperate partition run the above command on that, too
mount /dev/md1 boot
for i in `ls /dev/vg`; do mount /dev/vg/$i $i; done;

[edit] Stage 3 setup

• Go to new root:

cd /mnt/gentoo

• Get stage3 tarball and load it to /mnt/gentoo
• Unpack it:

tar xvjpf stage?-*.tar.bz2

[edit] Portage snapshot

• Get a portage snapshot and load it to /mnt/gentoo/usr
• Unpack it:

tar xjf portage*

[edit] Chrooting

• Copy nameserver info over:

cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf

• Copy RAID reference conf over:

cp -L /etc/mdadm.conf /mnt/gentoo/etc/mdadm.conf

• Mount proc and dev:

mount -t proc none /mnt/gentoo/proc
mount -o bind /dev /mnt/gentoo/dev

• Chroot to new root:

chroot /mnt/gentoo /bin/bash
env-update
source /etc/profile
export PS1="(chroot) $PS1"

[edit] System configuration

Follow Gentoo Handbook instructions and try to remember the following:

• compile raid1 support into kernel, this can be found under device-drivers->Multi-device support (RAID and LVM)
• if you use sata disks: compile scsi disk support into the kernel, as it is needed for autodetection
• compile Device Mapper support into the kernel (from lvm2-2.00.08 compiling LVM2 as a module won't work), this can also be found under device-drivers->Multi-device support (RAID and LVM)
• Install some userspace tools:

emerge -va sys-fs/lvm2 sys-fs/mdadm sys-fs/reiserfsprogs

(edit by DoubleHP 11th Feb 2007: sys-fs/raidtools are now deprecated; see bug #165917 )

Here are some system configuration files:

/dev/md1                /boot           ext3            noauto,noatime          1 2
/dev/md3                /               reiserfs        noatime                 0 0
/dev/md2                none            swap            sw                      0 0
/dev/vg/usr             /usr            reiserfs        noatime                 0 0
/dev/vg/var             /var            reiserfs        noatime                 0 0
/dev/vg/opt             /opt            reiserfs        noatime                 0 0
/dev/vg/tmp             /tmp            reiserfs        noatime                 0 0
/dev/vg/home            /home           reiserfs        noatime                 0 0
/dev/cdroms/cdrom0      /mnt/cdrom      iso9660         noauto,ro               0 0
none                    /proc           proc            defaults                0 0
none                    /dev/shm        tmpfs           defaults                0 0

As mentioned above, there is no need to use raid0 on swap partitions. Create both partitions as swap and put them in your fstab like this: "/dev/hda2 none swap sw,pri=0" Copy that line and use /dev/hdg2 for the other swap device. Like in this page. --DA

However, loss of a swap partition will cost you the system... Performance vs. stability. Your choice of course. --AK

NB! Fix /etc/lvm/lvm.conf for fast bootup (otherwise it will mess up with device scanning on boot):

#fill it with following
 devices {
    scan = [ "/dev/" ]
    filter = [ "a|/dev/md/|", "r|/dev/.*/|" ]
 }

[edit] Bootloader installation and configuration

• Install Grub:

emerge -va sys-boot/grub

• Invoke grub shell:

grub --no-floppy

• Within grub shell execute:

#setup MBR on /dev/hda
root (hd0,0)
setup (hd0)
#setup MBR on /dev/hdg
device (hd0) /dev/hdg
root (hd0,0)
setup (hd0)

NB! If one of the harddisks fails your motherboard bios has to be setup to boot from other harddisk as well.

• Create a /boot/grub/grub.conf file

 default 0
 timeout 8
 #Nice new 2.6 kernel
 title=Gentoo 2.6.16
 root (hd0,0)
 kernel /kernel-2.6.16-gentoo-r7 root=/dev/md3 md=3,/dev/hda3,/dev/hdg3

NB! If you use genkernel with initrd or initramfs. Kernel will not automatically setup raid devices and force all raid and lvm setup duty to initrd or initramfs. If you using genkernel you need too add some extra kernel options (dolvm2 and lvmraid=/dev/xxx,...) and make sure you comple kernel by use genkernel with --lvm2 option. If you also use LABEL instead of device name, then you must add --disklabel option too genkernel also.

NB2! If you using genkernel with a SCSI controller and boot drives you may need to add 'doscsi' to the kernel append parameters.

• compile kernel for lvm, software raid, dmraid and disklabel using genkernel

 emerge -va gentoo-sources
 emerge -va sys-kernel/genkernel
 genkernel --install --disklabel --lvm2 --dmraid all

• /boot/grub/grub.conf file for genkernel

 default 0
 timeout 8
 title Gentoo Linux 2006.0 [Default] genkernel-x86_64-2.6.16-gentoo-r7
 root (hd0,0)
 kernel /kernel-genkernel-x86_64-2.6.16-gentoo-r7 root=/dev/rd/0 init=/linuxrc real_root=LABEL=gt_root dolvm2 lvmraid=/dev/md1,/dev/md2,/dev/md3,/dev/md4
 initrd /initramfs-genkernel-x86_64-2.6.16-gentoo-r7

Edit by DoubleHP 11th Feb 2007: when using Raid 0 or 1 or 0+1 without LVM, you should not use root=LABEL= since those raid configuration will allow the kernel to read the indicated LABEL on some partition, and thus is likely to mount a partition instead of the raid chain. In some case, you may also try to omit 'dolvm2' and lvmraid=, and try to rely on the autoprobe function of the kernel, what will make your configuration more tolerant to hardware changes (disk swapping for example).

These are the kernel options I use for my md and lvm setup.--Bunkacid 11:51, 20 August 2007 (UTC)

part=/dev/hda3 lvmraid=/dev/md3 dodmraid dolvm2 root=/dev/ram0 xinit=/linuxrc real_root=/dev/mapper/Gentoo-root udev
                                   ^^ why do you have this here? --T3kkn0kr4t 11:35, 22 September 2007 (UTC)

On my system, without dodmraid, it will not initialize the device-mapper properly and therefore not boot up. --Bunkacid 22:47, 22 September 2007 (UTC)

[edit] Rebooting the system after install

umount /mnt/gentoo/var
umount /mnt/gentoo/usr
umount /mnt/gentoo/home
umount /mnt/gentoo/tmp
umount /mnt/gentoo/opt
umount /mnt/gentoo/proc
umount /mnt/gentoo/boot
umount /mnt/gentoo
vgchange -an
reboot

[edit] Short Version

umount /mnt/gentoo/*

Ignore the errors

umount /mnt/gentoo
vgchange -an
reboot

--mainframe 08:29, 26 Nov 2004 (GMT)

[edit] Troubleshooting

See Recovering RAID and LVM