KVM With Gentoo

We use as distribution a gentoo system. The KVM will be installed on a raid1 (mirror) with LVM2. We use the standard procedure to install the gentoo system. I start the installation from an already running system (you can even boot from a boot CD, USB stick or any other linux live system).

The system where I will install is located on /dev/sda. We will start with one harddisk, the second harddisk will be plugged into the system after it is running fine the raid system will mirror the data automatically. The harddisk must be completely empty.

Prepare the harddisk

modprobe raid1

Raid Device	/dev/sda	missing	Type
/dev/sda1	Bios boot partition	Bios boot partition	1MiB
/dev/md0	Raid for Gentoo-KVM	Raid for Gentoo-KVM	Raid-1 (mirror)

parted /dev/sda
mklabel gpt
(say yes if parted asks you)
mkpart boot1 1MiB 2MiB
mkpart kvm1 2MiB 100%
set 1 bios_grub on
set 2 raid on
quit

mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 missing
mdadm --grow /dev/md0 -b internal
pvcreate /dev/md0
vgcreate vg-kvm /dev/md0
lvcreate -L100M -nboot vg-kvm
lvcreate -L4GB -nswap vg-kvm
lvcreate -L30GB -nroot vg-kvm

mkfs.ext2 -L boot /dev/vg-kvm/boot
mkfs.ext4 -L root /dev/vg-kvm/root
mkswap /dev/vg-kvm/swap
swapon /dev/vg-kvm/swap

mount /dev/vg-kvm/root /mnt/gentoo/
mkdir /mnt/gentoo/boot
mount /dev/vg-kvm/boot /mnt/gentoo/boot/

Installation of Gentoo

Make sure you have the following use flags defined in /etc/portage/make.conf

USE="bindist mmx sse sse2 -ldap perl python lvm parted qemu virt-network avahi sasl audit nfs"
MAKEOPTS="-j5"
QEMU_SOFTMMU_TARGETS="i386 x86_64"
QEMU_USER_TARGETS="i386 x86_64"

Follow the normal gentoo install guide here: http://www.gentoo.org/doc/de/handbook/handbook-amd64.xml?part=1&chap=5

Some changes to the gentoo manual (all to be executed in the chroot environment):

Use lzo to compress kernel
Use Grub 2
Modify fstab to use labels
Build our own initramfs

echo "=sys-boot/grub-2*">>/etc/portage/package.keywords
echo "sys-boot/grub device-mapper">>/etc/portage/package.use
emerge -av grub lzop lvm2 mdadm genkernel
rc-update add lvm boot
rc-update add mdraid boot

Modify /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="dolvm domdadm intel_iommu=on,igfx_off,pass-through vga=791 splash=silent,theme:gentoo console=tty1 quiet libata.ignore_hpa=1 cgroup_enable=memory swapaccount=1"

To build the initramfs use:

genkernel --lvm --mdadm --iscsi --disklabel --install initramfs

Setup fstab:

blkid
/dev/loop0: TYPE="squashfs"
/dev/sda2: UUID="9338e0f6-0ed2-8112-a0e2-840b4d9616de" UUID_SUB="fa5fb4fe-6f65-efe6-5cd3-21e7e6c42976" LABEL="Gentoo-20121221:0" TYPE="linux_raid_member"
/dev/md0: UUID="PoGeVn-q9Sb-T7YD-Ysro-oRDG-pB7f-tXRWzr" TYPE="LVM2_member"
/dev/sdb: LABEL="PENDRIVE" UUID="1105-224F" TYPE="vfat"
/dev/mapper/vg--kvm-boot: LABEL="boot" UUID="251d9d9e-2c38-41db-938d-e25ef56afd3a" TYPE="ext2"
/dev/mapper/vg--kvm-swap: UUID="9d77c5f7-ebc8-4a9d-afa9-cd215489aae8" TYPE="swap"
/dev/mapper/vg--kvm-root: LABEL="root" UUID="7702989f-02c0-48dd-b640-3a921797096b" TYPE="ext4"

Now edit the /etc/fstab based on the information:

UUID="251d9d9e-2c38-41db-938d-e25ef56afd3a"             /boot           ext2            noatime  1 2
UUID="7702989f-02c0-48dd-b640-3a921797096b"             /               ext3            noatime         0 1
UUID="9d77c5f7-ebc8-4a9d-afa9-cd215489aae8"             none            swap            sw              0 0

Install GRUB 2

mkdir /boot/grub2
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/sda

Configure KVM

Install required software:

emerge -av qemu usbutils bridge-utils usermode-utilities iptables macchanger virt-manager
rc-update add libvirtd default
gpasswd -a qemu kvm

Enable KSM:

echo "echo 1 > /sys/kernel/mm/ksm/run" >/etc/local.d/ksm.start
chmod +x /etc/local.d/ksm.start

Create certificates:

mkdir -p /tmp/certs
cd /tmp/certs

certtool --generate-privkey > cakey.pem
echo "cn = FM-Data" >ca.info
echo "ca" >>ca.info
echo "cert_signing_key" >>ca.info
certtool --generate-self-signed --load-privkey cakey.pem --template ca.info --outfile cacert.pem

mkdir -p /etc/pki/CA
cp cacert.pem /etc/pki/CA

certtool --generate-privkey > serverkey.pem
echo "organization = FM-Data" >server.info
echo "cn = kvm.idefix.lan" >>server.info
echo "tls_www_server" >>server.info
echo "encryption_key" >>server.info
echo "signing_key" >>server.info
certtool --generate-certificate --load-privkey serverkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem --template server.info --outfile servercert.pem

mkdir -p /etc/pki/libvirt/private
cp serverkey.pem /etc/pki/libvirt/private/
cp servercert.pem /etc/pki/libvirt/

certtool --generate-privkey > clientkey.pem
echo "country = DE" >client.info
echo "state = Bayern" >>client.info
echo "locality = Fuerstenfeldbruck" >>client.info
echo "organization = FM-Data" >>client.info
echo "cn = client1" >>client.info
echo "tls_www_client" >>client.info
echo "encryption_key" >>client.info
echo "signing_key" >>client.info
certtool --generate-certificate --load-privkey clientkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem --template client.info --outfile clientcert.pem

cp clientkey.pem /etc/pki/libvirt/private/clientkey.pem
cp clientcert.pem /etc/pki/libvirt/clientcert.pem

rm *

Network bridge: Edit vi /etc/conf.d/net

bridge_br0="eth0"
brctl_br0="setfd 0 sethello 0 stp off"
config_br0="dhcp"

cd /etc/init.d/
ln -s net.lo net.br0
rc-update del net.eth0
rc-update add net.br0 default

Enable PCI passthrough:

echo "options kvm allow_unsafe_assigned_interrupts=1" >kvm_iommu_map_guest.conf

Add Second Harddisk and Build Raid1

Install sgdisk:

emerge -av sys-apps/gptfdisk

Copy the GPT partition schema from the original disk to the new disk. The already existing disk is /dev/sda and the new disk is /dev/sdf:

sgdisk -R=/dev/sdf /dev/sda
sgdisk -G /dev/sdf
sgdisk -c 1:"boot2" /dev/sdf
sgdisk -c 2:"kvm2" /dev/sdf

See disk and raid information:

cat /proc/mdstat
blkid
/dev/sdf1: PARTLABEL="boot2" PARTUUID="bea234c8-ce71-4c23-8e67-7faf5d8e50d1"
/dev/sdf2: PARTLABEL="kvm2" PARTUUID="054de73e-7cd6-4c94-860b-3eda62cabc58"

Add the new disk to the raid:

mdadm -a /dev/md126 /dev/disk/by-partuuid/054de73e-7cd6-4c94-860b-3eda62cabc58

Install boot manager:

grub2-install /dev/sdf

Install Gentoo as Guest

Download the install-amd64-minimal-20130207.iso and place it in the iso directory:

cd /mnt/isos/
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-iso/default/20130207/install-amd64-minimal-20130207.iso

Now start virt-manager and we create there the new virtual machine. I created a lvm partition with 30GB and mounted the iso we downloaded. Then follow the normal gentoo installation guide or continue here.

Prepare Harddisk

fdisk /dev/sda
n
p
1
<ENTER>
+100M
n
p
2
<ENTER>
<ENTER>
t
2
8e
a
1
w
pvcreate /dev/sda2
vgcreate vg-knx /dev/sda2
lvcreate -L4GB -nswap vg-knx
lvcreate -l+100%FREE -nroot vg-knx
mkfs.ext2 -L boot /dev/sda1
mkfs.ext4 -L root /dev/vg-knx/root
mkswap -L swap /dev/vg-knx/swap
swapon /dev/vg-knx/swap
mount /dev/vg-knx/root /mnt/gentoo/
mkdir /mnt/gentoo/boot
mount /dev/sda1 /mnt/gentoo/boot
cd /mnt/gentoo

Install Gentoo

links http://www.gentoo.org/main/en/mirrors.xml
(download stage3 in releases/autobuilds/current-stage3/default/..../stage3-amd64....tar.bz2)
q
tar xjpf stage3-*.tar.bz2
echo 'MAKEOPTS="-j2"' >>etc/portage/make.conf
mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
mirrorselect -i -r -o >> /mnt/gentoo/etc/portage/make.conf
cp -L /etc/resolv.conf /mnt/gentoo/etc/
mount -t proc none /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
source /etc/profile
export PS1="(chroot) $PS1"
mkdir /usr/portage
emerge-webrsync
eselect news list
eselect news read
eselect profile list
eselect profile set 3
emerge --sync --quiet
cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime
echo "Europe/Berlin" > /etc/timezone
emerge =gentoo-sources-3.0.35 eix genkernel syslog-ng vixie-cron dhcpcd grub vim
rc-update add syslog-ng default
rc-update add vixie-cron default
rc-update add sshd default

Prepare the kernel

vi /etc/genkernel.conf

SYMLINK="yes"
MAKEOPTS="-j2"
LVM="yes"
DISKLABEL="yes"
BOOTLOADER="grub"

Compile the kernel now with:

cd /usr/src/linux
genkernel all

Configure the system

vi /etc/fstab
LABEL="boot"            /boot           ext2            noatime         1 2
LABEL="root"            /               ext4            noatime         0 1
LABEL="swap"            none            swap            sw              0 0

Configure hostname:

vi /etc/conf.d/hostname
hostname="knx.idefix.lan"

Configure network:

vi /etc/conf.d/net
config_eth0="dhcp"

cd /etc/init.d
ln -s net.lo net.eth0
rc-update add net.eth0 default

Locale settings:

vi /etc/conf.d/keymaps
vi /etc/locale.gen
en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE ISO-8859-1
de_DE@euro ISO-8859-15
de_DE.UTF-8 UTF-8

locale-gen
vi /etc/env.d/02locale
LANG="de_DE.UTF-8"
LC_COLLATE="C"

env-update && source /etc/profile

Install grub

vi /boot/grub/grub.conf
default 0
timeout 0
splashimage=(hd0,0)/grub/splash.xpm.gz

title Gentoo Linux
root (hd0,0)
kernel /kernel real_root=/dev/vg-knx/root dolvm
initrd /initramfs

grep -v rootfs /proc/mounts > /etc/mtab
grub-install --no-floppy /dev/sda

Setup users

passwd
useradd -m -G users,wheel,audio -s /bin/bash idefix
passwd idefix

rm /stage3-*.tar.bz2*

Reboot the system

exit
cd
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -l /mnt/gentoo{/boot,/proc,}
reboot

Pass Cine-S2 to virtual machine

Find the device with:

lspci

we get:

...
04:00.0 Multimedia controller: Digital Devices GmbH Octopus LE DVB adapter
...

We search for the id of the card:

lspci -n |grep 04:00.0

We get:

04:00.0 0480: dd01:0003

Now we create a startup script to free the card before libvirt tries to start the virtual machine. For this we create a new start script and enable it:

cd /etc/init.d/
echo '#!/sbin/runscript' > pci_passthrough
echo "" >> pci_passthrough
echo "depend() {" >> pci_passthrough
echo "        before libvirtd" >> pci_passthrough
echo "}" >> pci_passthrough
echo "" >> pci_passthrough
echo "start () {" >> pci_passthrough
echo "        echo "dd01 0003" > /sys/bus/pci/drivers/pci-stub/new_id" >> pci_passthrough
echo "        echo 0000:04:00.0 >/sys/bus/pci/devices/0000\:04\:00.0/driver/unbind" >> pci_passthrough
echo "        echo 0000:04:00.0 >/sys/bus/pci/drivers/pci-stub/bind" >> pci_passthrough
echo "}" >> pci_passthrough
echo "" >> pci_passthrough
chmod +x pci_passthrough
rc-update add pci_passthrough default

If you reboot your system now, it frees the pci card with the given id before libvirt tries to start the virtual machines.

Snapshot and upgrade the guest, maybe roll back

We have now the following setup.

vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  boot           1   3   0 wz--n- 148,95g      0
  vg-kvm         1   7   0 wz--n- 232,75g 120,30g
  videostorage   1   1   0 wz--n-   4,00t  20,00g

  LV      VG           Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  root    boot         -wi-a---  30,00g
  swap    boot         -wi-a---   2,00g
  video1  boot         -wi-a--- 116,95g
  boot    vg-kvm       -wi-ao-- 100,00m
  freepbx vg-kvm       -wi-a---  19,53g
  iso     vg-kvm       -wi-ao--  10,00g
  knx     vg-kvm       -wi-ao--  29,30g
  root    vg-kvm       -wi-ao--  30,00g
  swap    vg-kvm       -wi-ao--   4,00g
  yavdr   vg-kvm       -wi-ao--  19,53g
  video0  videostorage -wi-ao--   3,98t

Create snapshot

We would like to create now a snapshot of the machine yavdr:

virsh shutdown yavdr

Wait till the virtual machine is shutdown completely. Check it with:

virsh list --all

Create the snapshot:

lvcreate -L4G -s -n /dev/vg-kvm/yavdr-snapshot /dev/vg-kvm/yavdr
virsh start yavdr

We can see the new snapshot now:

lvs vg-kvm
  LV             VG     Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  boot           vg-kvm -wi-ao-- 100,00m
  freepbx        vg-kvm -wi-a---  19,53g
  iso            vg-kvm -wi-ao--  10,00g
  knx            vg-kvm -wi-ao--  29,30g
  root           vg-kvm -wi-ao--  30,00g
  swap           vg-kvm -wi-ao--   4,00g
  yavdr          vg-kvm owi-aos-  19,53g
  yavdr-snapshot vg-kvm swi-a-s-   4,00g      yavdr    0,05

Roll back to snapshot

We tested now some stuff and we see it does not work. Stop the virtual machine:

virsh destroy yavdr

Rollback the lvm to the snapshot (this will destroy the snapshot):

lvconvert --merge /dev/vg-kvm/yavdr-snapshot

Start the VM again:

virsh start yavdr

Destroy the snapshot

If everything is fine, destroy the snapshot:

lvremove /dev/vg-kvm/yavdr-snapshot

Migrate XEN to KVM

Before you start to migrate xen to KVM, make sure that:

Block device will change from xvd? to vd?
- Change grub config and replace “root=/dev/xvda1 disk=/dev/xvda” by “root=/dev/vda1 disk=/dev/vda”
- Check fstab
serial device will change xvc0 to ttyS0
- Change grub config and remove console=xvc
- Edit /etc/inittab and comment line containing xvc0 and insert a line with tty1 instead
- Edit /etc/securetty and replace xvc0 with ttyS0
You system is fully bootable with a boot manager (like grub)
You have a full kernel installed

Change modules from:

xenblk xennet

virtio_blk virtio_pci virtio_net virtio_ballon

Copy KVM Image and Convert to KVM

Get a list of all XEN VMs:

xe vm-list

Copy one VM via ssh to another host:

xe vm-export uuid=<uuid> filename= | ssh -c blowfish -C <user>@<host> 'dd of=/path/vmname.xva'

Use the following script: http://www.robotics.net/wp-content/uploads/xenmigrate.py

Extract the the file with:

tar xvf vmname.xva

You should have a directory like:

---------- 1 root root   30251 Jan  1  1970 ova.xml
drwxr-xr-x 2 root root 1527808 Feb 24 11:10 Ref:5

Convert it with:

python xenmigrate.py --debug -–convert=Ref\:5 {image}.img

Now you can import the image with:

virt-install --name={image] --cpuset=auto --ram=2048 --network bridge=br123 --disk=/dev/vg/example --vnc --vnclisten=0.0.0.0 --noautoconsole --cdrom=/win2k8.iso