Xen-based Emulab virtual nodes
Table of Contents
Getting started
Emulab vnodes based on Xen virtual machines is still in an alpha stage, but is usable on the Utah Emulab. Since Xen vnode support is alpha quality, expect these instructions to be subject to sudden and radical change!
The executive summary is that you specify a Xen vnode by setting a node in your topology to be type pcvm (or pcvm3000, etc) and then choosing the correct OSID for the vnode and host. Each vnode will have around 200MB of RAM (on a pc3000) and an 8GB virtual disk. These are arbitrary constants that will be relaxed later. Shared filesystems are available in the vnodes. Vnodes can be connected with shaped virtual links, though expect lower fidelity emulations than with physical links. Each vnode has a control net connection, but they have unroutable IP addresses so you cannot communicate to/from vnodes and the outside (of Emulab) world.
Here is an example NS file for a three node, two link topology (node0 <=> node2 <=> node1):
set ns [new Simulator] source tb_compat.tcl # Nodes set node0 [$ns node] set node1 [$ns node] set node2 [$ns node] tb-set-node-os $node0 XEN-GUEST-F8-STD XEN3-UPDATE tb-set-node-os $node1 XEN-GUEST-F8-STD XEN3-UPDATE tb-set-node-os $node2 XEN-GUEST-F8-STD XEN3-UPDATE tb-set-hardware $node0 pcvm3000 tb-set-hardware $node1 pcvm3000 tb-set-hardware $node2 pcvm3000 # Links set link0 [$ns duplex-link $node0 $node2 100000.0kb 0.0ms DropTail] set link1 [$ns duplex-link $node1 $node2 100000.0kb 0.0ms DropTail] $ns rtproto Static $ns run
Details
Xen VMs current can only run a particular Fedora 8 Linux (2.6.18 kernel). The support is more general than that, but Fedora 8 is what we have now. Ideally, you would be able to load any Emulab OS image in a VM and have it Just Work, but that is harder than it might sound. The way you get Xen-based VMs as opposed to OpenVZ- or jail-based vnodes is via the OSID specified. So don't try to get all creative with the tb-set-node-os lines in the example--just do what we say.
Each Xen-based vnode will have an 8GB virtual disk (implemented as an LVM shadow volume on the physical host). Each VM is given an equal share of the physical host's RAM. Right now the calculation is based on 8 vnodes per pnode (regardless of how many vnodes you really have per pnode) with 256MB left for dom0. For a pc3000 (2GB) that works out to about 216MB per vnode.
Link shaping of virtual links is done using tc running in dom0.
Controlling Xen VM's
When dom0 has booted up you can ssh directly to it and use either Emulab commands or Xen commands to control the domU's. These need to be run with root priveleges (use sudo). Some helpful Emulab commands:
# Halt all vnodes
/usr/local/etc/emulab/bootvnodes -h
# Boot all vnodes
/usr/local/etc/emulab/bootvnodes -b
# Kill all vnodes (destroy virtual disks after halting)
/usr/local/etc/emulab/bootvnodes -k
Some helpful Xen commands:
# Start a domU
/usr/sbin/xm create <some-config-file>
# Halt a vm (clean shutdown)
/usr/sbin/xm shutdown <vm name>
# Kill a vm (unclean shutdown)
/usr/sbin/xm destroy <vm name>
# List all vm's
/usr/sbin/xm list
# Connect to the console of a vm
/usr/sbin/xm console <vm name>
Modifying Xen VM instances
Xen VM's use a configuration file to specify various attributes about their setup. Once dom0 has booted you can modify these configuration files as you like. The configuration files live in /var/emulab/vms/<vm-name>/xm.conf. Typical things to modify are the kernel and memory size. It is possible to modify the disk sizes as well but you have to use LVM commands directly and usually you have to create all the disks at once.
Any changes will not be persisted across reboots of dom0 (or use of the bootvnodes command above), so save your changes somewhere if you need them.
Windows on Xen
In theory, yes. In practice, not yet. Running Windows VMs will require machines with VTx technology, which we will soon have in the form of d710 machines.
Bugs
No doubt.
