Monthly Archives: October 2011

FUDCon Pune 2011

FUDcon (Fedora Users and Developers Conference) is happening in Pune, India. Plenty of interesting talks/demos/hack-fests are already scheduled. I’m giving two talks/demos. One on managing KVM hypervisor based guests via shell interface using Libvirt. The other, a working illustration of FreeIPA (with co-presenter shanks ). Will post my notes very soon.

Stay tuned.

Leave a comment

Filed under Uncategorized

Virtio Balloon in action — with native linux kvm tool

Over the weekend, for fun, I decided to tinker with the virtio ballon feature. For simplicity’s sake, I tested it with native linux kvm tool. I previously wrote about it here

If you want to get a quick refresher about virtio balloon concept, a while ago, Richard W Jones made an excellent post w/ nice illustration using images over here

Before I went ahead, I pulled the latest git, and applied a couple of patches from Sasha Levin(nlkt dev), which are on kvm mailing list, which fix the behaviour of kvm tool commands ‘stat’, ‘debug’ and a couple other things. Then I compiled the kvm tool, built the kernel bzImage, on a Fedora 16 box. Here is how it went:

First, setup a new rootfs(which will be stored under ~/.kvm-tools/default), and will mount your host file system in a read-only mode in the guest:


$ ./kvm setup default
  Info: Your new rootfs named default has been created.
You can now start it by running 'kvm run -d default'

$

Then, boot the kernel bzImage with the virtio ballon option. Once, booted, guest shell access over a serial console will be presented.
(Also, the below command line will allocate the guest memory, guest vcpu depending on the host configuration)


$ ./kvm run --balloon -k ../../arch/x86/boot/bzImage                                                                                                                                            
  # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-31362
.
.
.
[    1.505724] Bluetooth: BNEP filters: protocol multicast
[    1.508899] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[    1.511511] lib80211: common routines for IEEE802.11 drivers
[    1.513885] Installing 9P2000 support
[    1.517080] Registering the dns_resolver key type
[    1.518737]   Magic number: 15:85:793
[    1.520495] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.523497] EDD information not available.
[    1.537035] Sending DHCP requests ., OK
[    1.558038] IP-Config: Got DHCP answer from 192.168.33.1, my address is 192.168.33.15
[    1.560257] IP-Config: Complete:
[    1.561163]      device=eth0, addr=192.168.33.15, mask=255.255.255.0, gw=192.168.33.1,
[    1.563257]      host=192.168.33.15, domain=, nis-domain=(none),
[    1.564861]      bootserver=192.168.33.1, rootserver=0.0.0.0, rootpath=
[    1.567199] VFS: Unable to mount root fs via NFS, trying floppy.
[    1.574154] VFS: Mounted root (9p filesystem) on device 0:13.
[    1.576080] devtmpfs: mounted
[    1.578602] Freeing unused kernel memory: 512k freed
[    1.580356] Write protecting the kernel read-only data: 12288k
[    1.591640] Freeing unused kernel memory: 1916k freed
[    1.601083] Freeing unused kernel memory: 1436k freed
Mounting...
Starting '/bin/sh'...
sh-4.2# 

Get the guest current memory statistics:


$ ./kvm stat -m --all                                                                  


        *** Guest memory statistics ***

The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:495
The amount of memory not being used for any purpose (in bytes):438673408
The total amount of memory available (in bytes):449032192

$ 

List the current running guest instances:


$ ./kvm list -i
  PID GUEST
31362 guest-31362
$ 

Now, on a different terminal, let’s try to inflate/deflate the balloon. The current available memory, is roughly 448MB (Refer above. In bytes — 449032192)

To inflate, which means, to return memory from the guest, run the below command, which returns 10MB of memory back to the guest.


$ ./kvm balloon -n guest-31362 --inflate 10
$ 

Run the memory status again. Now, 10485760 bytes(10MB) is reduced from the guest and returned to the host. (which means, the virtio balloon is inflated)


$ ./kvm stat -m --all


        *** Guest memory statistics ***

The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:502
The amount of memory not being used for any purpose (in bytes):428339200
The total amount of memory available (in bytes):438546432

$ 

Now, let’s return back the memory to the guest from the host, by running the deflate command


$ ./kvm balloon -n guest-31362 --deflate 10
$ 

Again, re-check the memory status. 10485760 bytes(10MB) is now returned back to the guest from the host. (which means, the virtio balloon is inflated)


$ ./kvm stat -m --all                                                                  


        *** Guest memory statistics ***

The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:502
The amount of memory not being used for any purpose (in bytes):436846592
The total amount of memory available (in bytes):449032192

$ 

Now, like me, if you’re wondering what happens if you reduce the memory( inflate) by 10MB, and try to return back(deflate) 100 MB to the guest, it won’t let you.

Thanks to Sasha for helping me to fix some of the compile errors w/ newest gcc.

Leave a comment

Filed under Uncategorized

Configuring Certificate Chaining using mozilla nss(network security services)

Me and Shanks were trying to configure IPA server installation with an external CA set-up. For testing purpose, we planned to use certutil — part of mozilla nss-tools (Network Security Services) package. IPA generates a CSR(certificate signing request) called ipa.csr when you try to install it w/ the option –external-ca . The plan is to sign ipa.csr with an external CA(let’s say SubCA) which is signed by another CA(RootCA). So that there is a certificate chain of trust involved. After a bit of trial and error, this how it worked out:

To auto-generate a CSR (located /root/ipa.csr) to be signed by an external CA, while configuring IPA server, run:


$ ipa-server-install --external-ca

For illustration purpose, let’s create a temporary NSS db


$ certutil -N -d /var/tmp/testdb

Create a (self-signed)Root CA
Now, create a (self-signed)Root CA, and list the contents of the NSS(Network Security Services) db


$ certutil -S -n "IPA ROOTCA certificate" -s "cn=CAcert" -x -t "CT,," -m 1000 -v 120 -d .

$ certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

IPA ROOTCA certificate                                           CTu,u,u

Note: Under the ‘Trust Attributes’, the ‘u’ attribute indicates that corresponding private key is also available in that database. Keys can be listed by running # certutil -K -d /path/to/nssdb

Setup Subordinate CA
Create a CSR for the Subordinate CA which will be signed by the Root CA created in the previous step


$ certutil -R -s "cn=SUB-Certificate Authority, O=LAB.TEST.REDHAT.COM" -p "9323" -o ipasubca.req -d . -a

Now, sign this CSR with Master CA and output the certificate to a file


$ certutil -C -m 2346 -i ipasubca.req -o ipasubcacert.crt -c "IPA ROOTCA certificate" -d . -a

And, add this certificate to the NSS db, and provide a nick name to it.


$ certutil -A -d . -i ipasubcacert.crt -t "CTu,Cu,Cu" -n "IPA SUBCA certificate"

Now, list the certificates. (Both certs are listed and available)


$ certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

IPA ROOTCA certificate                                       CTu,u,u
IPA SUBCA certificate                                        CTu,Cu,Cu

Sign certs using the SubCA
First, export the ROOTCA cert into b64 ascii cert. So that we can later create an ascii chain ca cert file


$ certutil -L -d . -n "IPA ROOTCA certificate" -a > iparootca.crt

Create a CA chain file by concatenating iparootca.crt and ipasubcacert.crt into one single file.


$ cat iparootca.crt ipasubcacert.crt > chainca.crt

Let’s sign the ipa.csr using the SubCA configured previously.


$ certutil -C -m 2346 -i ipa.csr -o ipa.crt -c "IPA SUBCA certificate" -d . -a

Now, to install IPA server successfully using the external chainca set-up, provide this command-line


$ ipa-server-install --external_cert_file=ipa.crt --external_ca_file=/tmp/chainca.crt

Voila! IPA server successfully configured. [I didn’t include the complete, long stdout of ipa-server-install to keep this brief.]

Also note: Instead of the ipa.csr, you can sign any other CSRs(user,server,etc..) generated using either certutil or any other tool(like openssl) using the above RootCA/Subca set-up.

6 Comments

Filed under Uncategorized

pre-upgrade from f15 -> f16 – smooth as silk

For distro upgrades, I usually use pxe/tftpboot and do clean remote installs. So I thought I’ll give yum preupgrade a try for my remote test machines. I should also mention, this F15 machine is a minimal(@core) install.

Existing F15 install

 
# cat /etc/redhat-release 
Fedora release 15 (Lovelock)
 

Run the pre-upgrade command to upgrade to F16 branched pre-release

 
 preupgrade-cli "Fedora 16 Branched Pre-release (Verne)"
 

Everything went smooth.

Once, preupragde-cli is done, run the yum distribution-synchronization which synchronizes the installed package set with the latest packages available.

 
# yum distro-sync
 

Post preupgrade, let’s do some quick clean-up of dependency problems, orphans(installed pkgs which are not available from currently configured repos), duplicates, leaves(packages which are no longer needed by any other pkgs on the system)

 
 # package-cleanup --problems ; package-cleanup --orphans \
   package-cleanup --dupes ; package-cleanup --leaves
 

And reboot the host. There we go !

 
# cat /etc/redhat-release 
Fedora release 16 (Verne)
 

6 Comments

Filed under Uncategorized

Snapshotting with libvirt for qcow2 images

Libvirt 0.9.6 was recently out. Take a look at 0.9.5 changelog for truckload of features/bugfixes/cleanups(specifically snapshot related) from the libvirt team.

So, I grabbed the F14 srpm from Libvirt ftp, and made a quick Fedora koji scratch build of libvirt-0.9.6 for Fedora 15 and gave the snapshot features a whirl. Here it goes:

(Also noted below is some very useful discussion I had(on #virt, OFTC) with Eric Blake (Upstream/Red Hat’s Libvirt dev, very friendly guy.) on snapshots. It was way informative not to capture it.)

Context on types of snapshots
At the moment, snapshotting in KVM/QEMU/Libvirt land is supported primarily for QCOW2 disk images. I briefly discussed about Qcow2 previously here.

There are several different types of snapshots possible. Some idea on that:

Internal snapshot: A type of snapshot, where a single QCOW2 file will hold both the ‘saved state’ and the ‘delta’ since that saved point. ‘Internal snapshots’ are very handy because it’s only a single file where all the snapshot info. is captured, and easy to copy/move around the machines.

External snapshot: Here, the ‘original qcow2 file’ will be in a ‘read-only’ saved state, and the new qcow2 file(which will be generated once snapshot is created) will be the delta for the changes. So, all the changes will now be written to this delta file. ‘External Snapshots’ are useful for performing backups. Also, external snapshot creates a qcow2 file with the original file as its backing image, and the backing file can be /read/ in parallel with the running qemu.

VM State: This will save the guest/domain state to a file. So, if you take a snapshot including VM state, we can then shut off that guest and use the freed up memory for other purposes on the host or for other guests. Internally this calls qemu monitor’s ‘savevm’ command. Note that this only takes care of VM state(and not disk snapshot). To try this out:

 
#------------------------------------------------
# Memory before saving the guest f15vm3
$ free -m
             total       used       free     shared    buffers     cached
Mem:         10024       5722       4301          0        164       4445
-/+ buffers/cache:       1112       8911
Swap:            0          0          0
#------------------------------------------------
$ virsh list
 Id Name                 State
----------------------------------
  5 f15guest             running
  6 f15vm3               running
#------------------------------------------------
# Save the guest f15vm3 to a file 'foof15vm3'
$ virsh save f15vm3 foof15vm3
Domain f15vm3 saved to foof15vm3
#------------------------------------------------
# Now, f15vm3 is gracefully saved/shutdown.
$ virsh list
 Id Name                 State
----------------------------------
  5 f15guest             running
#------------------------------------------------
# Notice the RAM being freed
$ free -m
             total       used       free     shared    buffers     cached
Mem:         10024       5418       4605          0        164       4493
-/+ buffers/cache:        760       9263
Swap:            0          0          0
#------------------------------------------------
# Let's restore the guest back from the file 'foof15vm3'
$ virsh restore foof15vm3
Domain restored from foof15vm3
#------------------------------------------------
# List the status. f15vm3 is up and running.
$ virsh list
 Id Name                 State
----------------------------------
  5 f15guest             running
  7 f15vm3               running
#------------------------------------------------

For brevity, let’s try out internal disk snapshots where all the snapshot info. (like disk and VM state info) are stored in a single qcow2 file.
Virsh(libvirt shell interface to manage guests) has some neat options for snapshot supports. So, I’ve got an F15 guest (Qcow2 disk image).

Internal Disk Snapshots when the guest is online/running

For illustration purpose, let’s use a Fedora-15 guest called ‘f15guest’ .

$ virsh list
 Id Name                 State
----------------------------------
  4 f15guest             running

$ 

For clarity, ensure there are no prior snapshot instances around.

$ virsh snapshot-list f15guest
 Name                 Creation Time             State
------------------------------------------------------------

$ 

Before creating a snapshot, we need to create a snapshot xml file with 2 simple elements (name and description) if you need sensible name for the snapshot. Note that only these two fields are user settable. Rest of the info. will be filled by Libvirt.

$  cat /var/tmp/snap1-f15guest.xml
<domainsnapshot>
    <name>snap1-f15pki </name>
    <description>F15 system with dogtag pki packages </description>
</domainsnapshot>

$ 

Eric Blake noted that, the domainsnapshot xml file is optional now for ‘snapshot-create’ if you don’t need a description for the snapshot. And if it’s okay with libvirt generating the snapshot name for us. (More on this, refer below)

Now, I’m taking a snapshot while the ‘guest’ is running live. Here, Eric noted that, especially when running/live, the more RAM the guest has, and the more active the guest is modifying that RAM, the longer the it will take to create a snapshot. This was a guest was mostly an idle guest.

$ virsh snapshot-create f15guest /var/tmp/snap1-f15guest.xml
Domain snapshot snap1-f15pki  created from '/var/tmp/snap1-f15guest.xml'
$ 

While the snapshot-creation is in progress on the live guest, the state of the guest will be ‘paused’.

$ virsh list
 Id Name                 State
----------------------------------
  4 f15guest             paused

$ 

Once, the snapshot is created, list the snapshots of f15guest

$ virsh snapshot-list f15guest
 Name                 Creation Time             State
------------------------------------------------------------
 snap1-f15pki         2011-10-04 19:04:00 +0530 running

$

Internal snapshot while the guest is offline

For fun, I created another snapshot, but after shutting down the guest. Now, the snapshot creation is just instantaneous.

$ virsh list
 Id Name                 State
----------------------------------

$ virsh snapshot-create f15guest
Domain snapshot 1317757628 created

List the snapshots of ‘f15guest’ using virsh.

$ virsh snapshot-list f15guest
 Name                 Creation Time             State
------------------------------------------------------------
 1317757628           2011-10-05 01:17:08 +0530 shutoff
 snap1-f15pki         2011-10-04 19:04:00 +0530 running

To see some information about the VM size, snapshot info:

$ qemu-img info /export/vmimgs/f15guest.qcow2
image: /export/vmimgs/f15guest.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 3.2G
cluster_size: 65536
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         snap1-f15pki           1.7G 2011-10-04 19:04:00   32:06:34.974
2         1317757628                0 2011-10-05 01:17:08   00:00:00.000
$ 

To revert to a particular snapshot, virsh snapshot-revert domain snapshotname

Also, discussed with Eric, in what cases does virsh invoke Qemu’s ‘savevm‘ and ‘qemu-img snapshot -c‘ commands while creating different types of snapshots discussed earlier above. Here is the outline:

  • it uses ‘qemu-img snapshot -c‘ if the domain is offline and –disk-only was not specified
  • it uses qemu’s ‘savevm‘ if the domain is online and –disk-only was not specified
  • it uses qemu’s ‘snapshot_blkdev‘ if the domain is online and –disk-only is specified

(Note: –disk-only is an option to capture only ‘disk state’ but not VM state. This option is available w/ virsh ‘snapshot-create’ or ‘snapshot-create-as’ commands.)

Thanks Eric for the detail.

18 Comments

Filed under Uncategorized