Creating rapid thin-provisioned guests using QEMU backing files

Provisioning virtual machines very rapidly is highly desirable, especially, when deploying large number of virtual machines. With QEMU’s backing files concept, we can instantiate several clones, by creating a single base-image and then sharing it(read-only) across multiple guests. So that, these guests, when modified will write all their changes to their disk image

To exemplify:

Initially, let’s create a minimal Fedora 17 virtual guest (I used this script), and copy the resulting qcow2 disk image as base-f17.qcow2. So, base-f17.qcow2 has Fedora 17 on it, and is established as our base image. Let’s see the info of it

$ qemu-img info base-f17.qcow2
image: base-f17.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 5.0G
cluster_size: 65536
[root@localhost vmimages]# 

Now, let’s make use of the above F17 base image and try to instantiate 2 more Fedora 17 virtual machines, quickly. First, create a new qcow2 file(f17vm2-with-b.qcow2) using the base-f7.qcow2 as its backing-file:

$ qemu-img create -b /home/kashyap/vmimages/base-f17.qcow2 \
  -f qcow2 /home/kashyap/vmimages/f17vm2-with-b.qcow2
Formatting '/home/kashyap/vmimages/f17vm2-with-b.qcow2', fmt=qcow2 size=5368709120 backing_file='/home/kashyap/vmimages/base-f17.qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 

And now, let’s see some information about the just created disk image. (It can be noticed the ‘backing file’ attribute below pointing to our base image(base-f17.qcow2)

$ qemu-img info /home/kashyap/vmimages/f17vm2-with-b.qcow2
image: /home/kashyap/vmimages/f17vm2-with-b.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 196K
cluster_size: 65536
backing file: /home/kashyap/vmimages/base-f17.qcow2
[root@localhost vmimages]# 

Now, we’re set — our ‘f17vm2-with-b.qcow2‘ is ready to use. We can verify it in two ways:

  1. to quickly verify, we can invoke qemu-kvm (not recommended in production) — this will boot our new guest on stdio, and throws a serial console (NOTE: the base-f17.qcow2 had ‘console=tty0 console=ttyS0,115200′ on its kernel command line, so that it can provide serial console) —
    $ qemu-kvm -enable-kvm -m 1024 f17vm2-with-b.qcow2 -nographic
    
                              GNU GRUB  version 2.00~beta4
    
     +--------------------------------------------------------------------------+
     |Fedora Linux                                                              | 
     |Advanced options for Fedora Linux                                         |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          |
     |                                                                          | 
     +--------------------------------------------------------------------------+
    
          Use the ^ and v keys to select which entry is highlighted.      
          Press enter to boot the selected OS, `e' to edit the commands      
          before booting or `c' for a command-line.      
                                                                                   
                                                                                   
    Loading Linux 3.3.4-5.fc17.x86_64 ...
    Loading initial ramdisk ...
    [    0.000000] Initializing cgroup subsys cpuset
    .
    .
    .
    (none) login: root
    Password: 
    Last login: Thu Oct  4 07:07:54 on ttyS0
    $ 
    
  2. The other, more traditional way(so that libvirt could track it & can be used to manage the guest), is to copy a similar(F17) libvirt XML file, edit and update the name, uuid, disk path, mac-address, then define it, and start it via ‘virsh':
    $ virsh define f17vm2-with-b.xml
    $ virsh start f17vm2-with-b --console
    $  virsh list
     Id    Name                           State
    ----------------------------------------------------
     9     f17v2-with-b                  running
    

Now, let’s quickly check the disk-image size of our new thin-provisioned guest. It can be noticed, the size is quite thin (14Mb) — meaning, only the delta from the original backing file will be written to this image.

$ ls -lash f17vm2-with-b.qcow2
14M -rw-r--r--. 1 root root 14M Oct  4 06:30 f17vm2-with-b.qcow2
$

To instantiate our 2nd F17 guest(say f17vm3-with-b) — again, create a new qcow2 file(f17vm3-with-b.qcow2) with its backing file as our base image base-f17.qcow2 . And then, check the info of the disk image using ‘qemu-img’ tool.

#----------------------------------------------------------#
$ qemu-img create -b /home/kashyap/vmimages/base-f17.qcow2
 &nbsp -f qcow2 /home/kashyap/vmimages/f17vm3-with-b.qcow2
Formatting '/home/kashyap/vmimages/f17vm3-with-b.qcow2', fmt=qcow2 size=5368709120 backing_file='/home/kashyap/vmimages/base-f17.qcow2' encryption=off cluster_size=65536 lazy_refcounts=off 
#----------------------------------------------------------#
$ qemu-img info /home/kashyap/vmimages/f17vm3-with-b.qcow2
image: /home/kashyap/vmimages/f17vm3-with-b.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 196K
cluster_size: 65536
backing file: /home/kashyap/vmimages/base-f17.qcow2
$
#----------------------------------------------------------#

[it’s worth noting here that we’re pointing to the same base image, and multiple guests are using it as a backing file.]

Again check the disk image size of the thin-provisioned guest:

$ ls -lash f17vm3-with-b.qcow2
14M -rw-r--r--. 1 qemu qemu 14M Oct  4 07:18 f17vm3-with-b.qcow2

Goes without saying, the 2nd F17 guest also has a new XML file, defined w/ its unique attributes just like the 1st F17 guest.

$ virsh list
 Id    Name                           State
----------------------------------------------------
 9     f17vm2-with-b                  running
 10    f17vm3-with-b                  running

For reference sake, I’ve posted the xml file I’ve used for ‘f17vm3-with-b’ guest here

To summarize, by sharing a single, common base-image, we can quickly deploy multiple thin-provisioned virtual machines.


                      .----------------------.
                      | base-image-f17.qcow2 |
                      |                      |
                      '----------------------'
                         /       |         \
                        /        |          \
                       /         |           \
                      /          |            \
         .-----------v--.  .-----v--------.  .-v------------.
         | f17vm2.qcow2 |  | f17vm3.qcow2 |  | f17vmN.qcow2 |
         |              |  |              |  |              |
         '--------------'  '--------------'  '--------------'
            
About these ads

2 Comments

Filed under Uncategorized

2 responses to “Creating rapid thin-provisioned guests using QEMU backing files

  1. # modprobe nbd
    FATAL: Module nbd not found.
    # insmod nbd.ko
    insmod: can’t read ‘nbd.ko’: No such file or directory
    # uname -a
    Linux 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
    # rpm -qa | grep nbd
    nbd-2.9.20-7.el6.x86_64
    # locate nbd-server
    /usr/bin/nbd-server
    /usr/share/man/man1/nbd-server.1.gz
    /usr/share/man/man5/nbd-server.5.gz
    /var/cache/man/cat1/nbd-server.1.lzma
    # locate nbd-client
    /usr/sbin/nbd-client
    /usr/share/man/man8/nbd-client.8.gz
    # rpm -qa | grep qemu
    qemu-kvm-0.12.1.2-2.415.el6_5.7.x86_64
    qemu-kvm-tools-0.12.1.2-2.415.el6_5.7.x86_64
    qemu-guest-agent-0.12.1.2-2.415.el6_5.7.x86_64
    qemu-img-0.12.1.2-2.415.el6_5.7.x86_64
    gpxe-roms-qemu-0.9.7-6.10.el6.noarch

    what to do?
    Thanks

    • Seems like you’re using a very old Kernel.

      Works for me:

      $ sudo modprobe nbd
      $ echo $?
      0
      $ lsmod | grep nbd
      nbd                    17626  0 
      

      That’s the Kernel I’m using:

      $ uname -r
      3.13.4-200.fc20.x86_64+debug
      

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s