HOWTO Install Gentoo on an LVM2 root partition

From Gentoo Linux Wiki

This is where the community will put in a "HOWTO Install Gentoo on an LVM2 root partition" It will cover making an initrd and configuring boot scripts.

[edit] Needed Packages

• sys-apps/mkinitrd
• sys-fs/lvm2
• sys-fs/device-mapper

If you intend to use RAID, you will also need sys-fs/mdadm

[edit] Preamble

Ok, first ask the question, what do you need to mount a lvm partition?

1. Kernel with dm-mod loaded
2. Kernel with initrd support and CONFIG_BLK_DEV_RAM_SIZE > 4096
3. /dev/mapper/control Device hook for dm-mod
4. vgscan and vgchange
5. mount and the kernel drivers for the filesystem (ext2 must be built in for initrd to work)

[edit] Install

For our example we will use a single ide harddrive 40GB in size. The root filesystem will be reiserfs and the boot partition will be ext2.

We will use one volume group (system) and three logical volumes, usr, home and slash.

First, boot the gentoo liveCD and prepare the hard drive.

Prepare three partitions the first with id of 83 of at least 10MB (I usually make it 512MB to make it nice and round). The second will be your swap partion, mine is 512mb, use id 82. The third partition use the rest of the free space and use id 8e.

Now load the device mapper kernel module.

modprobe dm-mod

Now activate lvm (ignore any error messages about volume group not found at this point)

vgscan
vgchange -ay

Now make a physical volume on /dev/hda3

pvcreate /dev/hda3

Create a volume group on /dev/hda3, we will call it "system". The default physical extent size of 4MB is too little, so we set it to a more realistic 32MB here. We do this because our maximum VG size is determined by the size of our extents. Maximum VG size = Extent Size * 64K. E.g. 32MB * 64K = 2097152MB = 2048GB = 2TB. Think ahead; this number should be big enough to cover the largest size your array could possibly become...changing this later is a real pain (think backup / restore). On the other hand, we don't want to set this too large either because this is also the granularity with which we can allocate space to our Logical Volumes (i.e. if we used the maximum of 16GB, we'd only be able to allocate space in chunks of 16GB).

Note: the above restriction on extent size _only applies to LVM1_, there's no problem with 4MB extents and VG of any size in LVM2. Small extents in a very big VG will decrease the lvm tools performance, though. Source: vgcreate manpage.

vgcreate -s 32M system /dev/hda3

Note: only for LVM1, changing the extent size will possibly cause errors when extracting the portage snapshot.

Now make the logical volumes, we will leave some room to grow later.

lvcreate -L10G -nslash system
lvcreate -L10G -nusr system
lvcreate -L10G -nhome system

We have 9GB left, I'll show you how to use it later.
Let's format all out partitions now.

mkfs.ext2 /dev/hda1
mkswap /dev/hda2
mkfs.reiserfs /dev/system/slash
mkfs.reiserfs /dev/system/usr
mkfs.reiserfs /dev/system/home

Now let's mount these puppies!

swapon /dev/hda2
mkdir /gentoo
mount -t reiserfs /dev/system/slash /gentoo
mkdir /gentoo/boot
mount -t ext2 /dev/hda1 /gentoo/boot
mkdir /gentoo/usr
mount -t reiserfs /dev/system/usr /gentoo/usr
mkdir /gentoo/home
mount -t reiserfs /dev/system/home /gentoo/home

Download your gentoo stage tarball to /gentoo and install it.

cd /gentoo; tar -xvpjf ./stage{insertnamehere}.tar.bz2

You can alternatively extract the stage tarballs directly from the internet, without downloading it first.

cd /gentoo; wget http://www.yourfavoritemirror.org/gentoo/stageXYZ.tar.bz2 -O - | tar xvpjf -

Here we mount the proc filesystem for availability within the chroot env. This is need for many 2.6 kernel apps. We also bind the physical devices found by the live cd and provide it to the chroot env via the mount --bind method.

mkdir /gentoo/{dev,proc}
mount --bind /dev /gentoo/dev
mount -t proc /proc /gentoo/proc

Now chroot into your new install and continue as you would normally.

chroot /gentoo /bin/bash
env-update && source /etc/profile

Now continue as in Gentoo Handbook remember to enable the device mapper kernel module (dm-mod.ko).

[edit] /etc/fstab

You need to set this correctly.

/dev/mapper/system-slash   /           reiserfs        noatime,notail         0 0
/dev/mapper/system-usr     /usr        reiserfs        noatime                0 0
/dev/mapper/system-home    /home       reiserfs        noatime                0 0
/dev/hda2                  none        swap            sw                     0 0
/dev/hda1                  /boot       ext2            noauto,noatime         0 0

[edit] Make your initrd

You have three options here - each one gives a varying level of automation and control over the kernel and initrd configurations. You need an initrd, because you need userspace software to assemble the LVM/RAID volumes. Choose one of these sections based on your needs and preference.

[edit] The Easiest Way

In light of the gentoo way, Kernel/genkernel can be used to generate an initrd with lvm2 support built in. I built this with genkernel 3.1.0d and the following packages:

LVM2.2.00.25.tgz
device-mapper.1.00.17.tgz

[try running: "emerge lvm2" first, it may get lvm2 and device-mapper going right away. If portage complains about packages being blocked, try updating udev first]

cd /usr/src
rm linux
ln -sf linux-{version} linux

Now, all you need to do to generate the initrd with lvm2 support is the following:

genkernel --dmraid --lvm --splash=emergence initrd

The splash argument is for the splash theme when booting up, there are several options to choose from, but I chose emergence since it looks cool.

One last thing you need to do is tell the initrd that you want to use lvm2 and additionally raid if you use the lvm2 on top of a raid system. In grub, append dolvm2 to the kernel line in /boot/grub/grub.conf and be sure to set real_root=/dev/VOLUME_GROUP/ROOT_VOLUME where the VOLUME_GROUP AND ROOT_VOLUME are the devices for your system. In my case, I used /dev/vg/root.

Here is an example /boot/grub/grub.conf

#
# Sample boot menu configuration file
#

# Boot automatically after 10 secs.
timeout 10
# By default, boot the first entry.
default 0
# Fallback to the second entry.
fallback 1
splashimage=(hd0,0)/grub/splash.xpm.gz

# Gentoo Linux 2.6.9-r9 lvm
title  Gentoo Linux (2.6.9-r9 lvm2)
root   (hd0,0)
kernel /kernel-2.6.9-gentoo-r9 udev dolvm2 root=/dev/ram0 real_root=/dev/vg/root init=/linuxrc <other options>
initrd /initrd-2.6.9-gentoo-r9
# If you are using devfs instead of udev you should replace the kernel-line above with this line:
#kernel /kernel-2.6.9-gentoo-r9 devfs dolvm2 root=/dev/ram0 real_root=/dev/vg/root init=/linuxrc....

[edit] grub-install into mirror

If your boot partition is also on software raid, you will run into problems running grub-install. The problem is that grub-install does a 'df' to determine the real physical device corresponding with /boot. To get around this, you should edit /etc/mtab and change md0 to the 1st half of the mirror, run grub-install on the first half of the mirror, change it to the second half, run grub-install on the second half, then put /etc/mtab back. Assuming that /dev/md0 consists of /dev/sda1 and /dev/sdb1:

cp -p /etc/mtab /etc/mtab.orig
sed s:/dev/md0:/dev/sda1: /etc/mtab.orig > /etc/mtab
grub-install /dev/sda1
sed s:/dev/md0:/dev/sdb1: /etc/mtab.orig > /etc/mtab
grub-install /dev/sdb1
mv /etc/mtab.orig /etc/mtab

[edit] possible issue with above example

I am unfamiliar with grub, but dolvm2 did not work in lilo as of sys-kernel/genkernel-3.4.9_pre6. However, dolvm did work.

/etc/lilo.conf excerpt:

image=/boot/kernel-genkernel-x86-2.6.23-gentoo-r3
       label=2.6.23-r3
       root=/dev/ram0
       initrd=/boot/initramfs-genkernel-x86-2.6.23-gentoo-r3
       vga=798
       read-only
       append="dolvm real_root=/dev/sys/root init=/linuxrc ramdisk_size=8192"

[sireasoning]

[edit] Adding RAID with genkernel

If your lvm root device exists on a software md raid, you must tell your initrd about this so that the devices can be initialized before vgscan is run. To do this pass the line lvmraid=[md devices] either in the kernel line of your grub.conf or on the append line of your lilo.conf [fati]

[edit] Problems with this way

As of 2007-01-31, the genkernel way does not work out of the box (LVM on top of RAID), since it could not mount the root filesystem. It tried the following kernel arguments.

/boot/kernel-genkernel-x86-2.6.19-gentoo-r4 udev dodmraid dolvm2 root=/dev/ram0 init=/linuxrc lvmraid=/dev/md1 real_root=/dev/vg/root init=/linuxrc

The devices are found, but the userspace utilities (mdstart from busybox?) do not work. This seems to be a problem with newer versions of busybox. Try the 2nd easiest way if this one fails.

• Update 2007-10-03 Works fine, but don't forget to tell the initrd-script about the --dmraid option otherwise it seems that this would fail. The raid devices are found automatically as far the 0xfd-partition type is used.
• Make sure that your raid{0,1,etc} modules are loaded - they are not loaded by default with dmraid in genkernel-3.4.9. Either statically link them, or add doload=raid{0,1,etc} to your grub kernel parameters. Without them, busy box does not give any errors - it simply does not recognize the lvm inside the raid devices.

[edit] The Second Easiest Way (lvm2create_initrd)

This method uses the LVM2 script that automatically creates the initrd you need for LVM2. This way, you have more control over your kernel than genkernel gives you, and you don't have to make the initrd from scratch. The lvm2create_initrd script is available in the LVM2 source tree.

First, download the initrd script.

Next, as root, mount your /boot partition, if you need to:

mount /boot

Now, configure and compile your kernel however you want, keeping the following in mind:

• Device-mapper support should be compiled statically in to the kernel
• If you intend to use snapshots (one of the best features ever for doing backups), you should statically compile support for it into the kernel. NB: you can also do this as a module, but if you create a snapshot of your root partition, then do a reboot with that snapshot still available, your kernel will not be able to read the snapshot nor the source for the snapshot (i.e. root) causing it to panic...so, moral of the story: just compile it statically into your kernel and save yourself the trouble.

Device Drivers  --->
  [*] Multiple devices driver support (RAID and LVM) --->
    <*> Device mapper support
    <*> Snapshot target (EXPERIMENTAL)

• If you want to use Linux software RAID, compile code for that statically into the kernel.

Make sure to select the appropriate RAID type for your configuration. If in doubt, select all of these:

Device Drivers  --->
  [*] Multiple devices driver support (RAID and LVM) --->
    <*> RAID support
    <*> Linear (append) mode
    <*> RAID-0 (striping) mode 
    <*> RAID-1 (mirroring) mode
    <*> RAID-10 (mirrored striping) mode (EXPERIMENTAL)
    <*> RAID-4/RAID-5/RAID-6 mode

• Enable RAM disk. The RAM disk has to be large enough: the default of 4096Kb isn't enough (Seek beyond end of device errors on boot, not finding init). 8192 is almost always enough, unless you have a ton of built-in drivers (in that case, use 16384):

Device Drivers  --->
  Block Devices  --->
    <*> RAM disk Support
    (8192) Default RAM disk size (kbytes)

• Make sure static support for loopback devices is enabled as well.

Device Drivers  --->
  Block Devices  --->
    <*> Loopback device support 
 

• Enable initial ramdisk support.

   General setup --->
    [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
 

• And of course, basically anything you need to mount your root filesystem (ext2 for the initrd and other filesystems, device drivers) and operate a console while debugging (i.e. USB-Keyboard) has to be built statically.

Compile and install your kernel and modules. Go to the directory where the lvm2create_initrd script was downloaded. You have to specify the -M gentoo option to tell it which architecture to use. This will use the kernel pointed to by /usr/src/linux.

sh ./lvm2create_initrd -M gentoo

[edit] Specific kernel

If you have an specific kernel you would like to use, tell lvm2create_initrd about it by adding it to the command line, like this:

 sh ./lvm2create_initrd -M gentoo 2.6.9-gentoo-r4

[edit] lvm.conf

If you have an lvm.conf file already on your system, tell lvm2create_initrd about it by using the -c option, like this:

 sh ./lvm2create_initrd -M gentoo -c /etc/lvm/lvm.conf 2.6.9-gentoo-r4

The script should create the initrd, and move it to /boot, and give you some handy hints for setting up GRUB or LILO to boot your kernel and initrd. If all worked, make the necessary modifications to your bootloader.

[edit] Adding RAID

There are two options you need to pass to lvm2create_initrd to add software RAID support - that is, if your LVM2 root is on a RAID device. First, double check that the kernel support for the RAID level you are using is compiled into the kernel (this was done in an earlier step). Then, add the -r option to specify the name of the raid device you wish to have started, and the -R option to specify the path to the mdadm.conf file.

lvm2create_initrd -M gentoo -c /etc/lvm/lvm.conf -r /dev/md0 -R /etc/mdadm.conf 2.6.9-gentoo-r4

[edit] Create /initrd

Create the "initrd" directory under /:

mkdir -p /initrd

or you will get the error

Changing roots
\t*FAILED*
pivot_root: File not found (or something like this)

[edit] grub.conf

If you are using GRUB, you'll need something like this in grub.conf. Obviously, you'll need to replace your kernel, initrd and root volumes with the appropriate values for your system: VG is the name of the volume group containing the real root filesystem, and LV is the name of the logical volume within that volume group.

title Gentoo with LVM
    kernel /boot/vmlinuz-lvm2-2.6.9-gentoo-r4 root=/dev/ram0 lvm2root=/dev/VG/LV <other parameters>
    initrd /boot/initrd-lvm2-2.6.9-gentoo-r4.gz

Lilo users will want something like this. Again, modify the paths to the kernel, initrd, and lvm root volume as necessary:

image=/boot/vmlinuz-lvm2-2.6.9-gentoo-r4
     label="Gentoo_With_LVM"
     initrd=/boot/initrd-lvm2-2.6.9-gentoo-r4.gz
     append="root=/dev/ram0 lvm2root=/dev/VG/LV <other parameters>"

Ensure that your ramdisk size is sufficient to hold the initrd. Check your kernel configuration or override using the "ramdisk_size" (in kb) kernel parameter. The uncompressed size of your ramdisk can be found by passing the '-v' (verbose) option to lvm2create_initrd.

Then, continue on with this HOWTO, - you need to do the stuff in the "System Side" and "Cleanup" sections.

[edit] Potential Problems

[edit] bash fails on libncurses.so.5

On AMD64 platform I experienced this problem:

/bin/bash: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory

This appears to be a minor bug in Red Hat's script not liking our "lib64" directory. In case you haven't learned this yet, the initrd file is just a gzipped file containing an ext2 file system that the kernel loads into a ramfs. To solve this problem, I had to uncompress and mount the initrd file and make fix the "lib" directory.

cd /boot
gzip -d initrd-lvm2-2.6.21-gentoo-r1.gz
mkdir tmp
mount initrd-lvm2-2.6.21-gentoo-r1 tmp -o loop
cd tmp
mv lib/* lib64
rmdir lib
ln -s lib64 lib
cd ..
umount tmp
rmdir tmp
gzip initrd-lvm2-2.6.21-gentoo-r1

[edit] The Cowboy Way

If you don't like automation, and want to make your own initrd and kernel from scratch, this is the section for you!

CC="gcc -static" emerge lvm2
emerge mkinitrd
mkdir /initrd

pivotroot: pivot_root(/sysroot,/sysroot/initrd) failed: 2

My kernel is 2.6.9 use your own number for that...

mkinitrd --preload=dm-mod --with=dm-mod /boot/initrd-2.6.9.gz 2.6.9

Now mount the initrd.

mkdir /mnt/loop
gunzip /boot/initrd-2.6.9.gz
mount -o loop /boot/initrd-2.6.9 /mnt/loop

Now /mnt/loop has a miniture root filesystem. You need to copy a few utilites to it. Namley bash test mknod mkdir sed insmod

[edit] Static Executables

Build static versions of bash, busybox, lvm2 and mdadm into a different ROOT. Busybox and mdadm can build static versions using the static USE flag; the other two seem to work with CC="gcc -static", as follows:

mkdir /root/static
ROOT=/root/static USE="static -readline make-symlinks" emerge busybox mdadm
ROOT=/root/static CC="gcc -static" USE="-readline -gpm" emerge lvm2 bash

ROOT=/root/static USE="static -readline make-symlinks" LIBRARIES="-lsepol" ACCEPT_KEYWORDS="~amd64" emerge --ask -v busybox

cd /root/static
cp -a bin/bash bin/busybox bin/test /mnt/loop/bin
cp -a bin/mknod /bin/mkdir bin/sed /mnt/loop/bin
cp -a sbin/lvm sbin/mdadm /mnt/loop/sbin
cp -a /sbin/insmod.static /mnt/loop/sbin/insmod

(* mknod and mkdir are included in nash, which mkinitrd includes in the initrd. I guess in this case you should merge the below described devmap_mknod.sh into linuxrc?)

dd if=/boot/initrd-2.6.9 of=/dev/ram0
# above, /dev/ram0 represents any available/flushed ram device
# then resize, depending on the ramdisk size you have (to 6000 x 1K), then fsck, just in case
# your kernel default ramdisk size must support your image.
# otherwise you can specify it; and now ramdisk=XX in grub.conf is deprecated?
resize2fs -f /dev/ram0 6000
dd if=/dev/ram0 of=/boot/initrd-2.6.9 count=6000 ibs=1024
sync
e2fsck /boot/initrd-2.6.9

Make the script executable (chmod +x devmap_mknod.sh)

Add the script devmap_mknod.sh to /mnt/loop/bin

#! /bin/bash

# Startup script to create the device-mapper control device
# on non-devfs systems.
# Non-zero exit status indicates failure.
 
# These must correspond to the definitions in device-mapper.h and dm.h
DM_DIR="mapper"
DM_NAME="device-mapper"

set -e

DIR="/dev/$DM_DIR"
CONTROL="$DIR/control"

# Check for devfs, procfs
if test -e /dev/.devfsd ; then
        echo "devfs detected: devmap_mknod.sh script not required."
        exit
fi

if test ! -e /proc/devices ; then
        echo "procfs not found: please create $CONTROL manually."
        exit 1
fi

# Get major, minor, and mknod
MAJOR=$(sed -n 's/^ *\([0-9]\+\) \+misc$/\1/p' /proc/devices)
MINOR=$(sed -n "s/^ *\([0-9]\+\) \+$DM_NAME\$/\1/p" /proc/misc)

if test -z "$MAJOR" -o -z "$MINOR" ; then
        echo "$DM_NAME kernel module not loaded: can't create $CONTROL."
        exit 1
fi

mkdir -p -m 755 $DIR
test -e $CONTROL && rm -f $CONTROL

echo "Creating $CONTROL character device with major:$MAJOR minor:$MINOR."
mknod -m 600 $CONTROL c $MAJOR $MINOR

Edit /mnt/loop/linuxrc before the "pivot_root /sysroot /sysroot/initrd" line, the finished script should look like this.

#!/bin/bash

echo "Loading dm-mod.ko module"
insmod /lib/dm-mod.ko
echo Mounting /proc filesystem
mount -t proc /proc /proc
echo Creating block devices
mkdevices /dev
echo Creating root device
mkrootdev /dev/root
echo 0x0100 > /proc/sys/kernel/real-root-dev
echo "Making Device Mapper Nodes."
devmap_mknod.sh
echo "Scanning for volume groups."
lvm vgscan
lvm vgchange -ay
echo "Mounting root filesystem"
mount -o noatime --rw -t reiserfs /dev/mapper/system-slash /sysroot
pivot_root /sysroot /sysroot/initrd
umount /initrd/proc

Your initrd is now completed! Run this to finish up.

umount /mnt/loop
gzip /boot/initrd-2.6.9

[edit] System Side

You still have a little bit to do so gentoo can properly boot your system. Add these lines after "ebegin "Remounting root filesystem read-only (if necessary)"" in /etc/init.d/checkroot

vgscan --mknodes --ignorelockingfailure
vgchange -ay --ignorelockingfailure

[edit] Clean Up

Run these commands for good mesure.

emerge lvm2
emerge coreutils
vgchange -a n

Now install grub, and boot your new kernel with /boot/initrd-2.6.9 as your initrd, cross you fingers and hope for the best!

[edit] Links

• Official Gentoo LVM2 Guide
• Forum Post to troubleshoot Kernel Panics
• Official guide to md → lvm gentoo installs