We use as distribution a gentoo system. The KVM will be installed on a raid1 (mirror) with LVM2. We use the standard procedure to install the gentoo system. I start the installation from an already running system (you can even boot from a boot CD, USB stick or any other linux live system).

The system where I will install is located on /dev/sda. We will start with one harddisk, the second harddisk will be plugged into the system after it is running fine the raid system will mirror the data automatically. The harddisk must be completely empty.

modprobe raid1

parted /dev/sda
mklabel gpt
(say yes if parted asks you)
mkpart boot1 1MiB 2MiB
mkpart kvm1 2MiB 100%
set 1 bios_grub on
set 2 raid on
quit

mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 missing
mdadm --grow /dev/md0 -b internal
pvcreate /dev/md0
vgcreate vg-kvm /dev/md0
lvcreate -L100M -nboot vg-kvm
lvcreate -L4GB -nswap vg-kvm
lvcreate -L30GB -nroot vg-kvm

mkfs.ext2 -L boot /dev/vg-kvm/boot
mkfs.ext4 -L root /dev/vg-kvm/root
mkswap /dev/vg-kvm/swap
swapon /dev/vg-kvm/swap

mount /dev/vg-kvm/root /mnt/gentoo/
mkdir /mnt/gentoo/boot
mount /dev/vg-kvm/boot /mnt/gentoo/boot/

Make sure you have the following use flags defined in /etc/portage/make.conf

USE="bindist mmx sse sse2 -ldap perl python lvm parted qemu virt-network avahi sasl audit nfs"
MAKEOPTS="-j5"
QEMU_SOFTMMU_TARGETS="i386 x86_64"
QEMU_USER_TARGETS="i386 x86_64"

Follow the normal gentoo install guide here: http://www.gentoo.org/doc/de/handbook/handbook-amd64.xml?part=1&chap=5

Some changes to the gentoo manual (all to be executed in the chroot environment):

• Use lzo to compress kernel
• Use Grub 2
• Modify fstab to use labels
• Build our own initramfs

echo "=sys-boot/grub-2*">>/etc/portage/package.keywords
echo "sys-boot/grub device-mapper">>/etc/portage/package.use
emerge -av grub lzop lvm2 mdadm genkernel
rc-update add lvm boot
rc-update add mdraid boot

Modify /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="dolvm domdadm intel_iommu=on,igfx_off,pass-through vga=791 splash=silent,theme:gentoo console=tty1 quiet libata.ignore_hpa=1 cgroup_enable=memory swapaccount=1"

To build the initramfs use:

genkernel --lvm --mdadm --iscsi --disklabel --install initramfs

Setup fstab:

blkid
/dev/loop0: TYPE="squashfs"
/dev/sda2: UUID="9338e0f6-0ed2-8112-a0e2-840b4d9616de" UUID_SUB="fa5fb4fe-6f65-efe6-5cd3-21e7e6c42976" LABEL="Gentoo-20121221:0" TYPE="linux_raid_member"
/dev/md0: UUID="PoGeVn-q9Sb-T7YD-Ysro-oRDG-pB7f-tXRWzr" TYPE="LVM2_member"
/dev/sdb: LABEL="PENDRIVE" UUID="1105-224F" TYPE="vfat"
/dev/mapper/vg--kvm-boot: LABEL="boot" UUID="251d9d9e-2c38-41db-938d-e25ef56afd3a" TYPE="ext2"
/dev/mapper/vg--kvm-swap: UUID="9d77c5f7-ebc8-4a9d-afa9-cd215489aae8" TYPE="swap"
/dev/mapper/vg--kvm-root: LABEL="root" UUID="7702989f-02c0-48dd-b640-3a921797096b" TYPE="ext4"

Now edit the /etc/fstab based on the information:

UUID="251d9d9e-2c38-41db-938d-e25ef56afd3a"             /boot           ext2            noatime  1 2
UUID="7702989f-02c0-48dd-b640-3a921797096b"             /               ext3            noatime         0 1
UUID="9d77c5f7-ebc8-4a9d-afa9-cd215489aae8"             none            swap            sw              0 0

Install GRUB 2

mkdir /boot/grub2
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/sda

Install required software:

emerge -av qemu usbutils bridge-utils usermode-utilities iptables macchanger virt-manager
rc-update add libvirtd default
gpasswd -a qemu kvm

Enable KSM:

echo "echo 1 > /sys/kernel/mm/ksm/run" >/etc/local.d/ksm.start
chmod +x /etc/local.d/ksm.start

Create certificates:

mkdir -p /tmp/certs
cd /tmp/certs

certtool --generate-privkey > cakey.pem
echo "cn = FM-Data" >ca.info
echo "ca" >>ca.info
echo "cert_signing_key" >>ca.info
certtool --generate-self-signed --load-privkey cakey.pem --template ca.info --outfile cacert.pem

mkdir -p /etc/pki/CA
cp cacert.pem /etc/pki/CA

certtool --generate-privkey > serverkey.pem
echo "organization = FM-Data" >server.info
echo "cn = kvm.idefix.lan" >>server.info
echo "tls_www_server" >>server.info
echo "encryption_key" >>server.info
echo "signing_key" >>server.info
certtool --generate-certificate --load-privkey serverkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem --template server.info --outfile servercert.pem

mkdir -p /etc/pki/libvirt/private
cp serverkey.pem /etc/pki/libvirt/private/
cp servercert.pem /etc/pki/libvirt/

certtool --generate-privkey > clientkey.pem
echo "country = DE" >client.info
echo "state = Bayern" >>client.info
echo "locality = Fuerstenfeldbruck" >>client.info
echo "organization = FM-Data" >>client.info
echo "cn = client1" >>client.info
echo "tls_www_client" >>client.info
echo "encryption_key" >>client.info
echo "signing_key" >>client.info
certtool --generate-certificate --load-privkey clientkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem --template client.info --outfile clientcert.pem

cp clientkey.pem /etc/pki/libvirt/private/clientkey.pem
cp clientcert.pem /etc/pki/libvirt/clientcert.pem

rm *

Network bridge: Edit vi /etc/conf.d/net

bridge_br0="eth0"
brctl_br0="setfd 0 sethello 0 stp off"
config_br0="dhcp"

cd /etc/init.d/
ln -s net.lo net.br0
rc-update del net.eth0
rc-update add net.br0 default

Enable PCI passthrough:

echo "options kvm allow_unsafe_assigned_interrupts=1" >kvm_iommu_map_guest.conf

Install sgdisk:

emerge -av sys-apps/gptfdisk

Copy the GPT partition schema from the original disk to the new disk. The already existing disk is /dev/sda and the new disk is /dev/sdf:

sgdisk -R=/dev/sdf /dev/sda
sgdisk -G /dev/sdf
sgdisk -c 1:"boot2" /dev/sdf
sgdisk -c 2:"kvm2" /dev/sdf

See disk and raid information:

cat /proc/mdstat
blkid
/dev/sdf1: PARTLABEL="boot2" PARTUUID="bea234c8-ce71-4c23-8e67-7faf5d8e50d1"
/dev/sdf2: PARTLABEL="kvm2" PARTUUID="054de73e-7cd6-4c94-860b-3eda62cabc58"

Add the new disk to the raid:

mdadm -a /dev/md126 /dev/disk/by-partuuid/054de73e-7cd6-4c94-860b-3eda62cabc58 

Install boot manager:

grub2-install /dev/sdf

Download the install-amd64-minimal-20130207.iso and place it in the iso directory:

cd /mnt/isos/
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-iso/default/20130207/install-amd64-minimal-20130207.iso

Now start virt-manager and we create there the new virtual machine. I created a lvm partition with 30GB and mounted the iso we downloaded. Then follow the normal gentoo installation guide or continue here.

fdisk /dev/sda
n
p
1
<ENTER>
+100M
n
p
2
<ENTER>
<ENTER>
t
2
8e
a
1
w
pvcreate /dev/sda2
vgcreate vg-knx /dev/sda2
lvcreate -L4GB -nswap vg-knx
lvcreate -l+100%FREE -nroot vg-knx
mkfs.ext2 -L boot /dev/sda1
mkfs.ext4 -L root /dev/vg-knx/root
mkswap -L swap /dev/vg-knx/swap
swapon /dev/vg-knx/swap
mount /dev/vg-knx/root /mnt/gentoo/
mkdir /mnt/gentoo/boot
mount /dev/sda1 /mnt/gentoo/boot
cd /mnt/gentoo

links http://www.gentoo.org/main/en/mirrors.xml
(download stage3 in releases/autobuilds/current-stage3/default/..../stage3-amd64....tar.bz2)
q
tar xjpf stage3-*.tar.bz2
echo 'MAKEOPTS="-j2"' >>etc/portage/make.conf
mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
mirrorselect -i -r -o >> /mnt/gentoo/etc/portage/make.conf
cp -L /etc/resolv.conf /mnt/gentoo/etc/
mount -t proc none /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
source /etc/profile
export PS1="(chroot) $PS1"
mkdir /usr/portage
emerge-webrsync
eselect news list
eselect news read
eselect profile list
eselect profile set 3
emerge --sync --quiet
cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime
echo "Europe/Berlin" > /etc/timezone
emerge =gentoo-sources-3.0.35 eix genkernel syslog-ng vixie-cron dhcpcd grub vim
rc-update add syslog-ng default
rc-update add vixie-cron default
rc-update add sshd default

vi /etc/genkernel.conf

SYMLINK="yes"
MAKEOPTS="-j2"
LVM="yes"
DISKLABEL="yes"
BOOTLOADER="grub"

Compile the kernel now with:

cd /usr/src/linux
genkernel all

vi /etc/fstab
LABEL="boot"            /boot           ext2            noatime         1 2
LABEL="root"            /               ext4            noatime         0 1
LABEL="swap"            none            swap            sw              0 0

Configure hostname:

vi /etc/conf.d/hostname
hostname="knx.idefix.lan"

Configure network:

vi /etc/conf.d/net
config_eth0="dhcp"

cd /etc/init.d
ln -s net.lo net.eth0
rc-update add net.eth0 default

Locale settings:

vi /etc/conf.d/keymaps
vi /etc/locale.gen
en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE ISO-8859-1
de_DE@euro ISO-8859-15
de_DE.UTF-8 UTF-8

locale-gen
vi /etc/env.d/02locale
LANG="de_DE.UTF-8"
LC_COLLATE="C"

env-update && source /etc/profile

vi /boot/grub/grub.conf
default 0
timeout 0
splashimage=(hd0,0)/grub/splash.xpm.gz

title Gentoo Linux
root (hd0,0)
kernel /kernel real_root=/dev/vg-knx/root dolvm
initrd /initramfs

grep -v rootfs /proc/mounts > /etc/mtab
grub-install --no-floppy /dev/sda

passwd
useradd -m -G users,wheel,audio -s /bin/bash idefix
passwd idefix

rm /stage3-*.tar.bz2*

exit
cd
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -l /mnt/gentoo{/boot,/proc,}
reboot

Find the device with:

lspci

we get:

...
04:00.0 Multimedia controller: Digital Devices GmbH Octopus LE DVB adapter
...

We search for the id of the card:

lspci -n |grep 04:00.0

We get:

04:00.0 0480: dd01:0003

Now we create a startup script to free the card before libvirt tries to start the virtual machine. For this we create a new start script and enable it:

cd /etc/init.d/
echo '#!/sbin/runscript' > pci_passthrough
echo "" >> pci_passthrough
echo "depend() {" >> pci_passthrough
echo "        before libvirtd" >> pci_passthrough
echo "}" >> pci_passthrough
echo "" >> pci_passthrough
echo "start () {" >> pci_passthrough
echo "        echo "dd01 0003" > /sys/bus/pci/drivers/pci-stub/new_id" >> pci_passthrough
echo "        echo 0000:04:00.0 >/sys/bus/pci/devices/0000\:04\:00.0/driver/unbind" >> pci_passthrough
echo "        echo 0000:04:00.0 >/sys/bus/pci/drivers/pci-stub/bind" >> pci_passthrough
echo "}" >> pci_passthrough
echo "" >> pci_passthrough
chmod +x pci_passthrough
rc-update add pci_passthrough default

If you reboot your system now, it frees the pci card with the given id before libvirt tries to start the virtual machines.

We have now the following setup.

vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  boot           1   3   0 wz--n- 148,95g      0
  vg-kvm         1   7   0 wz--n- 232,75g 120,30g
  videostorage   1   1   0 wz--n-   4,00t  20,00g

  LV      VG           Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  root    boot         -wi-a---  30,00g
  swap    boot         -wi-a---   2,00g
  video1  boot         -wi-a--- 116,95g
  boot    vg-kvm       -wi-ao-- 100,00m
  freepbx vg-kvm       -wi-a---  19,53g
  iso     vg-kvm       -wi-ao--  10,00g
  knx     vg-kvm       -wi-ao--  29,30g
  root    vg-kvm       -wi-ao--  30,00g
  swap    vg-kvm       -wi-ao--   4,00g
  yavdr   vg-kvm       -wi-ao--  19,53g
  video0  videostorage -wi-ao--   3,98t

We would like to create now a snapshot of the machine yavdr:

virsh shutdown yavdr

Wait till the virtual machine is shutdown completely. Check it with:

virsh list --all

Create the snapshot:

lvcreate -L4G -s -n /dev/vg-kvm/yavdr-snapshot /dev/vg-kvm/yavdr
virsh start yavdr

We can see the new snapshot now:

lvs vg-kvm
  LV             VG     Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  boot           vg-kvm -wi-ao-- 100,00m
  freepbx        vg-kvm -wi-a---  19,53g
  iso            vg-kvm -wi-ao--  10,00g
  knx            vg-kvm -wi-ao--  29,30g
  root           vg-kvm -wi-ao--  30,00g
  swap           vg-kvm -wi-ao--   4,00g
  yavdr          vg-kvm owi-aos-  19,53g
  yavdr-snapshot vg-kvm swi-a-s-   4,00g      yavdr    0,05

We tested now some stuff and we see it does not work. Stop the virtual machine:

virsh destroy yavdr

Rollback the lvm to the snapshot (this will destroy the snapshot):

lvconvert --merge /dev/vg-kvm/yavdr-snapshot

Start the VM again:

virsh start yavdr

If everything is fine, destroy the snapshot:

lvremove /dev/vg-kvm/yavdr-snapshot