Creating a KVM playground · Ric Klaren's blog

Linux KVM provides a pretty neat virtualization solution on linux. It performs pretty well, and there is a quite a collection of tools like virt-builder, virt-install and libguestfs to help one automate all kinds of tasks.

If you have a bunch of virtual machines on KVM you often end up with or accessing them by IP or by maintaining a list of entries in /etc/hosts. Both of them are not ideal.

This post documents my local machine setup for KVM which improves on the vanilla setup.

My requirements

I hate remembering IP's so I want the setup to offer a simple demo domain with low maintenance. Which leads to this small list of requirements:

VM's get registered in a demo domain demo.box.
A host with hostname foo will be accessible as foo.demo.box.
Low maintenance.
Co-exists with libvirtd managed networks (so vagrant with KVM works).

The default libvirt behavior

Libvirt creates and manages a bunch of networks for the virtual machines it manages. By default there is one 'default' network that is used when you do not explicitly set a network while creating a VM. Let's see how the default network looks like that the VM's get hooked up to:

> virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>7250aca9-3e44-48f4-aeed-a0f090955ae9</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:32:7f:54'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

The network is configured to do NAT and offers DHCP in the 192.168.122.* IP range. The XML format is described here. Per network a dnsmasq instance is started that provides DHCP and DNS inside the NAT network.

Technically you could set up a caching dnsmasq instance that would cache all the separate dnsmasq instances. But in this setup those would refer to caching instance as well, creating a loop, which is probably not a good idea.

My solution is to disable the libvirtd managed dnsmasq instances and have one central caching dnsmasq to deal with DHCP for the virtual machines and DNS for both the host and the virtual machines. Added bonus is that dnsmasq will provide DNS caching as well.

To achieve this we need to:

Install the right software.
Configure the libvirt networks that we want to manage with the demo DNS.
Configure dnsmasq to manage a DHCP pool and DNS.
Configure the host (hypervisor) to use the central dnsmasq instance.

I'm using Fedora, a linux distribution that has served me pretty well over the years. At the time of writing I'm running Fedora 27. Almost all commands should be executed as root or via sudo. To create some of the configuration files in this post I use here documents for quick copy pasting. Commands are prepended with a > command output is not prepended. Of course the leading > does not need to be copied/typed.

Software installation

So let's get things installed by executing:

> dnf install libvirt dnsmasq
....
> systemctl enable libvirtd
....
> systemctl enable dnsmasq
....

You may want to install the management interface virt-manager as well, for the purpose of this post I will do things on the command line though.

Configuring the default network

Let us start by using virsh net-edit to change the definition of the default network. We will disable the DNS and DHCP by removing the DHCP range from the ip element and adding an element <dns enable='no'/>.

# virsh net-destroy stops the libvirt managed network, if it fails it is ok
> virsh net-destroy default
> virsh net-edit default
<network>
  <name>default</name>
  <uuid>7250aca9-3e44-48f4-aeed-a0f090955ae9</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:32:7f:54'/>
  <dns enable='no'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
  </ip>
</network>
> virsh net-start default

If you have other libvirt managed networks that you want to setup with the central DHCP/DNS then you apply the above procedure to them as well.

Configuring dnsmasq

Next prepare dnsmasq on the host (hypervisor). Edit /etc/dnsmasq.conf so that it looks like:

# Get upstream DNS settings from NetworkManager. It always manages a 
# resolv.conf in /run/NetworkManager/.
resolv-file=/run/NetworkManager/resolv.conf

# Queries in these domains are answered from /etc/hosts or DHCP only.
local=/demo.box/

# Only listen where we need to. If we need to listen on more interfaces 
# add them here. If we bind a wildcard here we will interfere with dnsmasq
# instances managed by libvirtd.
listen-address=192.168.122.1,127.0.0.1

# By using bind-dynamic there will be no clashes between this dnsmasq and the
# instances managed by libvirtd. More importantly it will also be robust
# against any of the listen-address interfaces not being up yet. Which is the
# case with the 192.168.122.* network, which is enabled later in the bootup
# sequence by libvirtd.
bind-dynamic

dhcp-range=192.168.122.10,192.168.122.254,12h

# when resolving foo append 'domain' to it.
expand-hosts
domain=demo.box

# Never forward plain names (without a dot or domain part) to upstream servers
domain-needed

# Never forward addresses in the non-routed address spaces.
# reverse lookups for private IP ranges (ie 192.168.x.x, etc) which are not 
# found in /etc/hosts or the DHCP leases file are answered with
# "no such domain"
bogus-priv

Next step is to reboot and verify that both dnsmasq and libvirtd started properly.

> systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor pre
   Active: active (running) since Wed 2018-03-28 23:30:36 EDT; 13h ago
....
> systemctl status dnsmasq
● dnsmasq.service - DNS caching server.
   Loaded: loaded (/usr/lib/systemd/system/dnsmasq.service; enabled; vendor pres
   Active: active (running) since Wed 2018-03-28 23:52:10 EDT; 13h ago
....

After verifying that they are properly running we can continue on with the next step.

Configure the host to use dnsmasq

A quick look at the /etc/resolv.conf shows that it is still managed by NetworkManager, this is not really what we want since it will keep on overwriting the /etc/resolv.conf.

> cat /etc/resolv.conf
# Generated by NetworkManager
search my-local.isp
nameserver 123.456.1.1

Let's take back control:

# Add a conf.d file so it survives upgrades.
> cat -> /etc/NetworkManager/conf.d/no-dns.conf <<EOF
[main]
dns = none
rc-manager = unmanaged
EOF
> systemctl restart NetworkManager

Note: technically it should be enough to add dns = none but I needed to add rc-manager = unmanaged as well.

And finally configure /etc/resolv.conf properly:

> cat -> /etc/resolv.conf <<EOF
search demo.box
nameserver 127.0.0.1
EOF
> reboot

After the reboot verify that the resolv.conf is still intact, and that DNS loopkups still work by visiting your favourite website.

Next create a test VM that requests a hostname and verify that it can be pinged:

> dnf install virt-install libguestfs-tools-c
....
> virt-builder cirros-0.3.5 \
  --format qcow2 \
  --output /var/lib/libvirt/images/cirros35.qcow2 \
  --size 50M \
  --memsize 512 \
  --network \
  --edit '/etc/hostname:s/.+/cirros35/' \
  --edit '/etc/cirros-init/config:s/DATASOURCE_LIST=.+/DATASOURCE_LIST="nocloud"/' \
  --edit '/boot/grub/menu.lst:s/console=ttyS0//'
[   3.8] Downloading: http://libguestfs.org/download/builder/cirros-0.3.5.xz
[   3.8] Planning how to build this image
[   3.8] Uncompressing
[   4.3] Converting raw to qcow2
[   4.4] Resizing container (but not filesystems) to expand the disk to 50.0M
[   4.5] Opening the new disk
[  20.5] Setting a random seed
[  20.5] Editing: /etc/hostname
[  20.6] Editing: /etc/cirros-init/config
[  20.6] Editing: /boot/grub/menu.lst
[  20.7] Setting passwords
virt-builder: warning: password: using insecure md5 password encryption for 
guest of type cirros version 0.3.
If this is incorrect, use --password-crypto option and file a bug.
virt-builder: Setting random password of root to LIrxJeVHePi0yEGx
[  21.1] Finishing off
                   Output file: /var/lib/libvirt/images/cirros35.qcow2
                   Output size: 50.0M
                 Output format: qcow2
            Total usable space: 22.2M
                    Free space: 5.3M (23%)
> virt-install --name=cirros35 \
  --ram=512 \
  --vcpus=1 \
  --disk path=/var/lib/libvirt/images/cirros35.qcow2,format=qcow2,bus=virtio \
  --import \
  --network network:default \
  --noautoconsole
WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.

Starting install...
Domain creation completed.
# Allow the domain to start up and:
> ping cirros35.demo.box
PING cirros35.demo.box (192.168.122.17) 56(84) bytes of data.
64 bytes from cirros35.demo.box (192.168.122.17): icmp_seq=1 ttl=64 time=1.40 ms

Enjoy!