Nova’s way of using a disk image when it boots a guest for first time

Let’s see how Nova — the compute component of OpenStack — uses a disk image from the time it was initially imported into Glance (OpenStack’s repository for virtual machine disk images) till a new virtual machine instance is booted and running.

I started by downloading a Fedora-17 disk image from here
(Keep a note of the size of the image — 241 MB )

[tuser1@interceptor ~(keystone_user1)]$ ls -lash f17-x86_64-openstack-sda.qcow2 
241M -rw-rw-r--. 1 tuser1 tuser1 241M Jan 13  2012 f17-x86_64-openstack-sda.qcow2
[tuser1@interceptor ~(keystone_user1)]$ 

Import the above Fedora 17 disk image into Glance:

[tuser1@interceptor ~(keystone_admin)]$ glance image-create --name="fedora-17" --is-public=true \
--disk-format=qcow2 --container-format bare < f17-x86_64-openstack-sda.qcow2

(Note that I have to source Keystone admin credentials to list images from Glance).

Let’s list the current images in Glance:

tuser1@interceptor ~(keystone_admin)]$ glance image-list
| ID                                   | Name      | Disk Format | Container
Format | Size       | Status |
| 1e6292f9-82bd-4cdb-969e-c863cb1c6692 | fedora-17 | qcow2       | bare
| 251985920  | active |
| acc4c853-9153-4e80-b3c8-e253451ae983 | rhel63    | qcow2       | bare
| 1074135040 | active |
[tuser1@interceptor ~(keystone_admin)]$ 

List current running Nova instances :

[tuser1@interceptor ~(keystone_user1)]$ nova list
| ID                                   | Name      | Status | Networks          |
| 08d616a9-87a1-4c0d-b986-7d6aa5ed6780 | fedora-t1 | ACTIVE | net1=ww.xx.yyy.zz |
| 3e487977-37e8-4f26-9443-d65ecbdf83c9 | fedora-t2 | ACTIVE | net1=ww.xx.yyy.zz |
| 48d9e518-a91f-48db-9d9b-965b243e7113 | fedora-t4 | ACTIVE | net1=ww.xx.yyy.zz |
[tuser1@interceptor ~(keystone_user1)]$ 

Let’s also list the guests running using libvirt’s virsh :

[tuser1@interceptor ~(keystone_user1)]$ sudo virsh list
 Id    Name                           State
 12    instance-0000000c              running
 13    instance-0000000d              running
 22    instance-00000012              running

[tuser1@interceptor ~(keystone_user1)]$ 

Find the block device in use for one of the running instances to examine it further — instance-0000000c, in this case:

[tuser1@interceptor ~(keystone_user1)]$ sudo virsh domblklist instance-0000000c
Target     Source
vda        /var/lib/nova/instances/instance-0000000c/disk

[tuser1@interceptor ~(keystone_user1)]$ 

Let’s get information about the disk in use by the above nova instance; specifically, find its backing file:

[tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/instance-0000000c/disk
image: /var/lib/nova/instances/instance-0000000c/disk
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 149M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20

Now, get information about the “backing file” used by the overlay (used by the running Nova instance above):

[tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
image: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 740M
[tuser1@interceptor ~(keystone_user1)]$ 

It’s worth noting here that the original disk image initially uploaded into glance was a qcow2 image. However, the base image being used by Nova is a raw disk image.

From this, we can see: When booting a virtual machine instance for the first time, Nova does a few things:

  1. Make a copy of the original qcow2 disk image from Glance, convert it into a ‘raw’ sparse image & make it a base image (its location – /var/lib/nova/instances/_base) — Reason for turning non-raw to raw, refer “Background info” section below.
  2. Expands the size of the base image to 20GB (because, I used the m1.small flavour from Nova, when initially booting the image.)
  3. Use this base image & instantiate a copy on write overlay (qcow2) to boot the nova instance.

To demonstrate this post, I used Red Hat OpenStack Folsom, RHEL 6.4, single node, running these services — nova, glance, keystone, cinder; using KVM as the underlying hypervisor.

Background info:

[1] From nova’s git log, the conversion from non ‘raw’ images to ‘raw’ is tracked down to this commit

commit ff9d353b2f4fee469e530fbc8dc231a41f6fed84
Author: Scott Moser 
Date:   Mon Sep 19 16:57:44 2011 -0400

    convert images that are not 'raw' to 'raw' during caching to node
    This uses 'qemu-img' to convert images that are not 'raw' to be 'raw'.
    By doing so, it
     a.) refuses to run uploaded images that have a backing image reference
         (LP: #853330, CVE-2011-3147)
     b.) ensures that when FLAGS.use_cow_images is False, and the libvirt
         xml written specifies 'driver_type="raw"' that the disk referenced
         is also raw format. (LP: #837102)
     c.) removes compression that might be present to avoid cpu bottlenecks
         (LP: #837100)
    It does have the negative side affect of using more space in the case where
    the user uploaded a qcow2 (or other advanced image format) that could have
    been used directly by the hypervisor.  That could, later, be remedied by
    another 'qemu-img convert' being done to the "preferred" format of the

[2] Pádraig Brady pointed out (thank you !) this bz — to note further reasons for converting images from non raw to raw.

As an aside, more informaion on disk image size allocation/ performance improvements upstream –

UPDATE : Pádraig Brady writes in excellent detail about the life of an openstack libvirt image — describing Openstack —

About these ads


Filed under Uncategorized

3 responses to “Nova’s way of using a disk image when it boots a guest for first time

  1. Pratt G

    Hi ,
    Can we configure fedora version less then 17 to work on openstack ?
    If yes,how we can do that?Please specify in detail.

    Thanks in advance.

    Pratt G

    • Fedora 17 is outdated. Try newer ones.
      Specifics take time, post your notes if you get something working.

      • Pratt G

        Thanks for your quick reply but I mean to say is can we boot a image of fedora version less then 17 on openstack ?
        If yes,how can I do that?
        Please specify in detail.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s