We use as distribution a gentoo system. The KVM will be installed on a raid1 (mirror) with LVM2. We use the standard procedure to install the gentoo system. I start the installation from an already running system (you can even boot from a boot CD, USB stick or any other linux live system).
The system where I will install is located on /dev/sda. We will start with one harddisk, the second harddisk will be plugged into the system after it is running fine the raid system will mirror the data automatically. The harddisk must be completely empty.
Prepare the harddisk
modprobe raid1| Raid Device | /dev/sda | missing | Type |
|---|---|---|---|
| /dev/sda1 | Bios boot partition | Bios boot partition | 1MiB |
| /dev/md0 | Raid for Gentoo-KVM | Raid for Gentoo-KVM | Raid-1 (mirror) |
parted /dev/sda
mklabel gpt
(say yes if parted asks you)
mkpart boot1 1MiB 2MiB
mkpart kvm1 2MiB 100%
set 1 bios_grub on
set 2 raid on
quitmdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 missing
mdadm --grow /dev/md0 -b internal
pvcreate /dev/md0
vgcreate vg-kvm /dev/md0
lvcreate -L100M -nboot vg-kvm
lvcreate -L4GB -nswap vg-kvm
lvcreate -L30GB -nroot vg-kvmmkfs.ext2 -L boot /dev/vg-kvm/boot
mkfs.ext4 -L root /dev/vg-kvm/root
mkswap /dev/vg-kvm/swap
swapon /dev/vg-kvm/swapmount /dev/vg-kvm/root /mnt/gentoo/
mkdir /mnt/gentoo/boot
mount /dev/vg-kvm/boot /mnt/gentoo/boot/Installation of Gentoo
Make sure you have the following use flags defined in /etc/portage/make.conf
USE="bindist mmx sse sse2 -ldap perl python lvm parted qemu virt-network avahi sasl audit nfs"
MAKEOPTS="-j5"
QEMU_SOFTMMU_TARGETS="i386 x86_64"
QEMU_USER_TARGETS="i386 x86_64"Follow the normal gentoo install guide here: http://www.gentoo.org/doc/de/handbook/handbook-amd64.xml?part=1&chap=5
Some changes to the gentoo manual (all to be executed in the chroot environment):
- Use lzo to compress kernel
- Use Grub 2
- Modify fstab to use labels
- Build our own initramfs
echo "=sys-boot/grub-2*">>/etc/portage/package.keywords
echo "sys-boot/grub device-mapper">>/etc/portage/package.use
emerge -av grub lzop lvm2 mdadm genkernel
rc-update add lvm boot
rc-update add mdraid bootModify /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="dolvm domdadm intel_iommu=on,igfx_off,pass-through vga=791 splash=silent,theme:gentoo console=tty1 quiet libata.ignore_hpa=1 cgroup_enable=memory swapaccount=1"To build the initramfs use:
genkernel --lvm --mdadm --iscsi --disklabel --install initramfsSetup fstab:
blkid
/dev/loop0: TYPE="squashfs"
/dev/sda2: UUID="9338e0f6-0ed2-8112-a0e2-840b4d9616de" UUID_SUB="fa5fb4fe-6f65-efe6-5cd3-21e7e6c42976" LABEL="Gentoo-20121221:0" TYPE="linux_raid_member"
/dev/md0: UUID="PoGeVn-q9Sb-T7YD-Ysro-oRDG-pB7f-tXRWzr" TYPE="LVM2_member"
/dev/sdb: LABEL="PENDRIVE" UUID="1105-224F" TYPE="vfat"
/dev/mapper/vg--kvm-boot: LABEL="boot" UUID="251d9d9e-2c38-41db-938d-e25ef56afd3a" TYPE="ext2"
/dev/mapper/vg--kvm-swap: UUID="9d77c5f7-ebc8-4a9d-afa9-cd215489aae8" TYPE="swap"
/dev/mapper/vg--kvm-root: LABEL="root" UUID="7702989f-02c0-48dd-b640-3a921797096b" TYPE="ext4"Now edit the /etc/fstab based on the information:
UUID="251d9d9e-2c38-41db-938d-e25ef56afd3a" /boot ext2 noatime 1 2
UUID="7702989f-02c0-48dd-b640-3a921797096b" / ext3 noatime 0 1
UUID="9d77c5f7-ebc8-4a9d-afa9-cd215489aae8" none swap sw 0 0Install GRUB 2
mkdir /boot/grub2
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/sdaConfigure KVM
Install required software:
emerge -av qemu usbutils bridge-utils usermode-utilities iptables macchanger virt-manager
rc-update add libvirtd default
gpasswd -a qemu kvmEnable KSM:
echo "echo 1 > /sys/kernel/mm/ksm/run" >/etc/local.d/ksm.start
chmod +x /etc/local.d/ksm.startCreate certificates:
mkdir -p /tmp/certs
cd /tmp/certs
certtool --generate-privkey > cakey.pem
echo "cn = FM-Data" >ca.info
echo "ca" >>ca.info
echo "cert_signing_key" >>ca.info
certtool --generate-self-signed --load-privkey cakey.pem --template ca.info --outfile cacert.pem
mkdir -p /etc/pki/CA
cp cacert.pem /etc/pki/CA
certtool --generate-privkey > serverkey.pem
echo "organization = FM-Data" >server.info
echo "cn = kvm.idefix.lan" >>server.info
echo "tls_www_server" >>server.info
echo "encryption_key" >>server.info
echo "signing_key" >>server.info
certtool --generate-certificate --load-privkey serverkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem --template server.info --outfile servercert.pem
mkdir -p /etc/pki/libvirt/private
cp serverkey.pem /etc/pki/libvirt/private/
cp servercert.pem /etc/pki/libvirt/
certtool --generate-privkey > clientkey.pem
echo "country = DE" >client.info
echo "state = Bayern" >>client.info
echo "locality = Fuerstenfeldbruck" >>client.info
echo "organization = FM-Data" >>client.info
echo "cn = client1" >>client.info
echo "tls_www_client" >>client.info
echo "encryption_key" >>client.info
echo "signing_key" >>client.info
certtool --generate-certificate --load-privkey clientkey.pem --load-ca-certificate cacert.pem --load-ca-privkey cakey.pem --template client.info --outfile clientcert.pem
cp clientkey.pem /etc/pki/libvirt/private/clientkey.pem
cp clientcert.pem /etc/pki/libvirt/clientcert.pem
rm *Network bridge: Edit vi /etc/conf.d/net
bridge_br0="eth0"
brctl_br0="setfd 0 sethello 0 stp off"
config_br0="dhcp"cd /etc/init.d/
ln -s net.lo net.br0
rc-update del net.eth0
rc-update add net.br0 defaultEnable PCI passthrough:
echo "options kvm allow_unsafe_assigned_interrupts=1" >kvm_iommu_map_guest.confAdd Second Harddisk and Build Raid1
Install sgdisk:
emerge -av sys-apps/gptfdiskCopy the GPT partition schema from the original disk to the new disk. The already existing disk is /dev/sda and the new disk is /dev/sdf:
sgdisk -R=/dev/sdf /dev/sda
sgdisk -G /dev/sdf
sgdisk -c 1:"boot2" /dev/sdf
sgdisk -c 2:"kvm2" /dev/sdfSee disk and raid information:
cat /proc/mdstat
blkid
/dev/sdf1: PARTLABEL="boot2" PARTUUID="bea234c8-ce71-4c23-8e67-7faf5d8e50d1"
/dev/sdf2: PARTLABEL="kvm2" PARTUUID="054de73e-7cd6-4c94-860b-3eda62cabc58"Add the new disk to the raid:
mdadm -a /dev/md126 /dev/disk/by-partuuid/054de73e-7cd6-4c94-860b-3eda62cabc58 Install boot manager:
grub2-install /dev/sdfInstall Gentoo as Guest
Download the install-amd64-minimal-20130207.iso and place it in the iso directory:
cd /mnt/isos/
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-iso/default/20130207/install-amd64-minimal-20130207.isoNow start virt-manager and we create there the new virtual machine. I created a lvm partition with 30GB and mounted the iso we downloaded. Then follow the normal gentoo installation guide or continue here.
Prepare Harddisk
fdisk /dev/sda
n
p
1
<ENTER>
+100M
n
p
2
<ENTER>
<ENTER>
t
2
8e
a
1
w
pvcreate /dev/sda2
vgcreate vg-knx /dev/sda2
lvcreate -L4GB -nswap vg-knx
lvcreate -l+100%FREE -nroot vg-knx
mkfs.ext2 -L boot /dev/sda1
mkfs.ext4 -L root /dev/vg-knx/root
mkswap -L swap /dev/vg-knx/swap
swapon /dev/vg-knx/swap
mount /dev/vg-knx/root /mnt/gentoo/
mkdir /mnt/gentoo/boot
mount /dev/sda1 /mnt/gentoo/boot
cd /mnt/gentooInstall Gentoo
links http://www.gentoo.org/main/en/mirrors.xml
(download stage3 in releases/autobuilds/current-stage3/default/..../stage3-amd64....tar.bz2)
q
tar xjpf stage3-*.tar.bz2
echo 'MAKEOPTS="-j2"' >>etc/portage/make.conf
mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
mirrorselect -i -r -o >> /mnt/gentoo/etc/portage/make.conf
cp -L /etc/resolv.conf /mnt/gentoo/etc/
mount -t proc none /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
source /etc/profile
export PS1="(chroot) $PS1"
mkdir /usr/portage
emerge-webrsync
eselect news list
eselect news read
eselect profile list
eselect profile set 3
emerge --sync --quiet
cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime
echo "Europe/Berlin" > /etc/timezone
emerge =gentoo-sources-3.0.35 eix genkernel syslog-ng vixie-cron dhcpcd grub vim
rc-update add syslog-ng default
rc-update add vixie-cron default
rc-update add sshd defaultPrepare the kernel
vi /etc/genkernel.confSYMLINK="yes"
MAKEOPTS="-j2"
LVM="yes"
DISKLABEL="yes"
BOOTLOADER="grub"Compile the kernel now with:
cd /usr/src/linux
genkernel allConfigure the system
vi /etc/fstab
LABEL="boot" /boot ext2 noatime 1 2
LABEL="root" / ext4 noatime 0 1
LABEL="swap" none swap sw 0 0Configure hostname:
vi /etc/conf.d/hostname
hostname="knx.idefix.lan"Configure network:
vi /etc/conf.d/net
config_eth0="dhcp"cd /etc/init.d
ln -s net.lo net.eth0
rc-update add net.eth0 defaultLocale settings:
vi /etc/conf.d/keymaps
vi /etc/locale.gen
en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE ISO-8859-1
de_DE@euro ISO-8859-15
de_DE.UTF-8 UTF-8
locale-gen
vi /etc/env.d/02locale
LANG="de_DE.UTF-8"
LC_COLLATE="C"
env-update && source /etc/profileInstall grub
vi /boot/grub/grub.conf
default 0
timeout 0
splashimage=(hd0,0)/grub/splash.xpm.gz
title Gentoo Linux
root (hd0,0)
kernel /kernel real_root=/dev/vg-knx/root dolvm
initrd /initramfsgrep -v rootfs /proc/mounts > /etc/mtab
grub-install --no-floppy /dev/sdaSetup users
passwd
useradd -m -G users,wheel,audio -s /bin/bash idefix
passwd idefix
rm /stage3-*.tar.bz2*Reboot the system
exit
cd
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -l /mnt/gentoo{/boot,/proc,}
rebootPass Cine-S2 to virtual machine
Find the device with:
lspciwe get:
...
04:00.0 Multimedia controller: Digital Devices GmbH Octopus LE DVB adapter
...We search for the id of the card:
lspci -n |grep 04:00.0We get:
04:00.0 0480: dd01:0003Now we create a startup script to free the card before libvirt tries to start the virtual machine. For this we create a new start script and enable it:
cd /etc/init.d/
echo '#!/sbin/runscript' > pci_passthrough
echo "" >> pci_passthrough
echo "depend() {" >> pci_passthrough
echo " before libvirtd" >> pci_passthrough
echo "}" >> pci_passthrough
echo "" >> pci_passthrough
echo "start () {" >> pci_passthrough
echo " echo "dd01 0003" > /sys/bus/pci/drivers/pci-stub/new_id" >> pci_passthrough
echo " echo 0000:04:00.0 >/sys/bus/pci/devices/0000\:04\:00.0/driver/unbind" >> pci_passthrough
echo " echo 0000:04:00.0 >/sys/bus/pci/drivers/pci-stub/bind" >> pci_passthrough
echo "}" >> pci_passthrough
echo "" >> pci_passthrough
chmod +x pci_passthrough
rc-update add pci_passthrough defaultIf you reboot your system now, it frees the pci card with the given id before libvirt tries to start the virtual machines.
Snapshot and upgrade the guest, maybe roll back
We have now the following setup.
vgs
VG #PV #LV #SN Attr VSize VFree
boot 1 3 0 wz--n- 148,95g 0
vg-kvm 1 7 0 wz--n- 232,75g 120,30g
videostorage 1 1 0 wz--n- 4,00t 20,00g LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root boot -wi-a--- 30,00g
swap boot -wi-a--- 2,00g
video1 boot -wi-a--- 116,95g
boot vg-kvm -wi-ao-- 100,00m
freepbx vg-kvm -wi-a--- 19,53g
iso vg-kvm -wi-ao-- 10,00g
knx vg-kvm -wi-ao-- 29,30g
root vg-kvm -wi-ao-- 30,00g
swap vg-kvm -wi-ao-- 4,00g
yavdr vg-kvm -wi-ao-- 19,53g
video0 videostorage -wi-ao-- 3,98tCreate snapshot
We would like to create now a snapshot of the machine yavdr:
virsh shutdown yavdrWait till the virtual machine is shutdown completely. Check it with:
virsh list --allCreate the snapshot:
lvcreate -L4G -s -n /dev/vg-kvm/yavdr-snapshot /dev/vg-kvm/yavdr
virsh start yavdrWe can see the new snapshot now:
lvs vg-kvm
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
boot vg-kvm -wi-ao-- 100,00m
freepbx vg-kvm -wi-a--- 19,53g
iso vg-kvm -wi-ao-- 10,00g
knx vg-kvm -wi-ao-- 29,30g
root vg-kvm -wi-ao-- 30,00g
swap vg-kvm -wi-ao-- 4,00g
yavdr vg-kvm owi-aos- 19,53g
yavdr-snapshot vg-kvm swi-a-s- 4,00g yavdr 0,05Roll back to snapshot
We tested now some stuff and we see it does not work. Stop the virtual machine:
virsh destroy yavdrRollback the lvm to the snapshot (this will destroy the snapshot):
lvconvert --merge /dev/vg-kvm/yavdr-snapshotStart the VM again:
virsh start yavdrDestroy the snapshot
If everything is fine, destroy the snapshot:
lvremove /dev/vg-kvm/yavdr-snapshot| Aug 14 | Migrate XEN to KVM | 1 min read |
| Dec 16 | Atmel | 2 min read |
| Dec 16 | Compiling a kernel | 1 min read |
| Dec 16 | Distcc and ccache | 4 min read |
| Dec 16 | LVM2 | 3 min read |