HOWTO Virtual Xen Servers and Gentoo

From Gentoo Linux Wiki

It has been suggested that this article or section be merged into HOWTO Xen and Gentoo. (Discuss)

[edit] Abstract

This article describes the installation of several virtual servers on a single server. These servers can be used to run webservers, database servers or any other kind. It will use XEN as a virtualization plattform and is based on a real world setup example. It also covers creation, copying and administration of the virtual servers.

[edit] Prerequisites

It is assumed that you have experience with Gentoo and Xen, and all systems you want to install on your servers. This HOWTO will only show how to set up the bare bone system. You should read the Xen and Gentoo HOWTO for setting up a Xen0 domain. You should read the LVM and RAID1 with Gentoo if you want to use the same setup as in this article.

[edit] Setup

The example system will consist of a single CPU server with two S-ATA hard disks. The discs will be synced by scripts introduced in this article. Xen servers will live in img files and share the portage tree. The Xen0 will be on the boot partition, which is synced to the second disk when you think it is necessary or by a script.

[edit] Dom0 Configuration

You need the following packages to install XEN:

 # emerge xen-sources
 # emerge xen-tools

After that, change into to source directory and copy the source tree, once for Dom0, which we will use now, and one for the DomU's, if needed.

 # cd /usr/src
 # cp -a linux-VERSION-xen linux-VERSION-xen-domU
 # mv linux-VERSION-xen linux-VERSION-xen-dom0

In order to run several machines, you need a lot of loop devices (something that doesn't come as standard). I'll show how to configure the loop device as a module (for static linking change the max_loop value in drivers/block/loop.c) (Note: drivers/block/loop.c reads: "Maximum number of loop devices when compiled is selectable by passing max_loop=<1-256> to the kernel on boot).

 # cd /usr/src/linux-VERSION-xen-dom0
 # make menuconfig (make sure Device Drivers->Block Devices->Loopback support is marked as <M>)
 # make; make modules; make install; make modules_install;
 # echo "loop max_loop=256" >> /etc/modules.autoload.d/kernel-2.6

After reboot a lot of (XEN) stuff should appear in the boot messages.

[edit] Network configuration

This section is only for the 3.0.4 p1 release of Xen on Gentoo. I had a lot of problems while migrating, mostly because things were more complicated in 3.0.2. First configure your system as you would for a non-bridged environment, i.e. set your eth0 like in

config_eth0=( "217.147.216.40 netmask 255.255.255.240 broadcast 217.147.216.47" )
routes_eth0=(
        "default via 217.147.216.46"
)

Then make sure that you have it in your init.d and that it is called at boot time:

# ls -l /etc/init.d/net.eth0
lrwxrwxrwx 1 root root 6 May 15 10:10 /etc/init.d/net.eth0 -> net.lo

# cat /etc/conf.d/rc  | grep RC_PLUG_SERVICES
RC_PLUG_SERVICES="net.*"   ####IMPORTANT: Remove the "!" before the net.*

[edit] Firewall configuration

The setup is such that each server has its own firewall configuration. For this, the DomU's have to be in something you could call a DMZ. Because they have a nice explanation (look here), I used the Shorewall system. You can follow that installation 1:1, just make sure your bridge is called the same (specified in xend-config.xsp) as you specify in the shorewall files (in the example xenbr0). Also make sure you have your local network specified correctly in shorewall/rules

[edit] DomUs

[edit] Intro

I have set up a webserver, a mailserver, two nameservers (altough this is a bit ridiculous, as probably both will fail the same time), a fileserver, a database server an intranet server and a game server. I will briefly show how I did this.

[edit] Creation of a generic domU

For all of the above servers, I started with a generic server, which we will create now. First, create a directory to keep all the Xen stuff. Also, create a mountpoint for the images

 # mkdir -p /xen/images; mkdir /mnt/mp1

Then you can use this script to create the images needed:

#!/bin/sh

dd if=/dev/zero of=./$1_root.img bs=1024k count=2048
dd if=/dev/zero of=./$1_var.img bs=1024k count=5120
dd if=/dev/zero of=./$1_swap.img bs=1024k count=1024

mkfs.ext2 $1_root.img
mkfs.ext2 $1_var.img
mkswap $1_swap.img

exit 0

[edit] Administration

[edit] Automating Server Creation

There is a tool called domi (apparently from SuSE), which can set up servers for you. I couldn't find a lot of documentation about it, but it looks promising and probably a lot better then what I'm proposing. However, it downloads complete distros. The method proposed here needs a generic server that you have set up above, so you won't have to pull the distro from the web each time for the server (maybe domi supports that as well?)

[edit] Giving Priority

I heuristically found out that domains with a lot of load can block others. I didn't want that and with very little knowledge what I'm actually doing I found out that this script ususally fixed it (if someone from the Xen developers ever reads this, I appologize for my not understanding schedulers...)

xm sched-sedf Domain-0 -w 1
xm sched-sedf generic -w 1
xm sched-sedf nameserver -w 1
xm sched-sedf nameserver2 -w 1
xm sched-sedf mailserver -w 1
xm sched-sedf webserver -w 1
xm sched-sedf intranet -w 1
xm sched-sedf db -w 1
xm sched-sedf fileserver -w 1

Of course you will have to add any other servers you have installed to this. I also added this script to the /etc/init.d/xendomains script (put it right before the closing } from the start{ section), but this is not mandatory.

[edit] Installing new kernels on DomU

If you want to update your kernels on domUs, then you will have to shut down that instance. After that you build the new kernel in dom0, mount the root partition of the domU you want to update and install it onto that mount point by using the INSTALL_MOD_PATH environment variable.

 # xm shutdown your_instance
 # mount -o loop /xen/images/intranet_root.img /mnt/mp1
 # cd /usr/src/your_new_kernel_src
 # export INSTALL_MOD_PATH="/mnt/mp1/"
 # make modules_install
 # umount /mnt/mp1

[edit] Sharing Files between domains

[edit] Samba

In order to share volumes between domains, they have to be shared like you would share drives between seperate computers, which involves network file systems. The first choice would have been nfs, but I couldn't get that working (XenU domains would block upon connecting). So I used samba, because it doesn't need that much kernel stuff as openafs. If you're willing to experiment, or using a later version of Xen, other choices might be better. To set up the Samba Server and Client see the HOWTO Samba on Gentoo

[edit] NFS

1. export your file system by modifying /etc/exports, and adding a line of "/path 192.168.7.50(async,rw)". DO NOT use hostnames unless they are DNS resolvable (/etc/hosts will not work). Otherwise your NFS will hang (or block) upon connecting. Instead, use IP addresses, if the hostname is not resolvable through DNS.
2. start your NFS service
3. add a line to /etc/fstab on your client. The host name should be DNS resolvable, or /etc/hosts resolvable. The line should be like "hostname:/path /path nfs noatime". Then, go ahead and mount it.

[edit] Date/Time in DomU

All domains should have the same time, if you run ntp-client in dom0. If, however, this is not the case (as it is with my setup for unknown reasons), you will have to run ntp-client on all guests. For this to work, you will need to set /proc/sys/xen/independent_wallclock to 1, i.e.

 echo 1 > /proc/sys/xen/independent_wallclock

and set clock to local in /etc/conf.d/clock

# /etc/conf.d/clock

# Set CLOCK to "UTC" if your system clock is set to UTC (also known as
# Greenwich Mean Time).  If your clock is set to the local time, then
# set CLOCK to "local".  Note that if you dual boot with Windows, then
# you should set it to "local".

CLOCK="local"

 

If you want to have this set each time you boot the domU, add it in /etc/init.d/clock

        ...
        elif is_xenU_sys ; then
                echo 1 > /proc/sys/xen/independent_wallclock
                TBLURB="xen"
                fakeit=1
        ...

Another approach... Using openntpd in dom0 (without setting independent_wallclock) the domUs will drift. The following, run periodically from dom0 will resync the domUs. There are probably a hundred reasons not to do this, but it does mean I only have one instance of openntpd to maintain and find memory for, instead of many.

date -s "`date`"

[edit] Links

Setting up a nameserver HOWTO