Emulab FAQ: PlanetLab: Setting up a private PlanetLab inside Emulab

It is now possible to set up a private PlanetLab inside Emulab.

The current setup is based on the MyPLC package. For more information on MyPLC go to http://www.planet-lab.org/doc/myplc.

A private PlanetLab inside Emulab (plab-in-emulab) experiment is the same as a normal experiment except for a few special things. In your NS file you declare one node to act as PlanetLab Central (PLC) using the commands:

tb-set-node-os $plc PLAB-PLC-v42
tb-set-node-plab-role $plc plc

You then also need to declare which nodes will act as PlanetLab nodes using the command:

tb-set-node-plab-role $plab node

for each such node.

Since MyPLC does not seem to support binding some key services (such as the bootserver and PLC API server) to multiple interfaces, you can setup your private PlanetLab to communicate either on the Emulab control network, which is publicly routable, or over experimental LAN interfaces. Currently, the default is the control network. If you wish to specify that the private PLC and associated nodes should communicate over Emulab experimental LANs or links, you must first setup those LANs and links, and then use the tb-set-node-plab-plcnet command to bind those nodes' PLC communication to the specified LAN or link. For instance, to force PLC communication to take place over a simple experimental LAN:

set innerPlanetLabLan [$ns make-lan "$plc $node(1) $node(2)" 100Mb 0ms]

tb-set-node-plab-plcnet $plc $innerPlanetLabLan
tb-set-node-plab-plcnet $node(1) $innerPlanetLabLan
tb-set-node-plab-plcnet $node(2) $innerPlanetLabLan

You can bind each node to any link or LAN you choose; however, you must ensure that any nodes you wish to boot successfully can reach the private PLC. All communications with PLC will take place over innerPlanetLabLan, with two exceptions. First, you will still be able to access the private PLC website from the fully-qualified hostname of your PLC nodes (i.e., https://plc.expname.projname.emulab.net). Second, to automate the PlanetLab node boot process, the nodes boot via TFTP from the PLC over the control network interface. After TFTP communication, however, all communication between the node and PLC will occur over the experimental LAN, if you have selected one.

You can use our template to create an NS file, or you can use the one below:

set ns [new Simulator]
source tb_compat.tcl

set num_pcs 2

set plc [$ns node]
tb-set-node-os $plc PLAB-PLC-v42
tb-set-node-plab-role $plc plc
tb-set-hardware $plc pc3000

set lan_string ""

for {set i 1} {$i <= $num_pcs} {incr i} {
  set n($i) [$ns node]
  tb-set-node-plab-role $n($i) node
  append lan_string "$n($i) "
}

set lan [$ns make-lan $lan_string 100Mb 0ms]

$ns rtproto Static
$ns run

Emulab currently provides several MyPLC images. PLAB-PLC-v42 supports PlabetLab 4.2 and represents our best effort to emulate current production PlanetLab functionality (it is currently built using the MyPLC 4.2-16.planetlab.2008.07.08 from PlanetLab 4.2-rc18). Unfortunately, pc3000 cannot be used for the PlanetLab nodes since they will not boot (it is okay to use them for PLC). Thus, you will need to specify some other hardware type for the PlanetLab nodes using tb-set-hardware, pc850 and pc2400c2 are supported. You can also use tb-make-soft-vtype to specify either pc850 or pc2400c2. For example:

...
tb-make-soft-vtype plabpc {pc850 pc2400c2}
...
for {set i 1} {$i <= $num_pcs} {incr i} {
  set n($i) [$ns node]
  tb-set-node-plab-role $n($i) node
  tb-set-hardware $n($i) plabpc 
  append lan_string "$n($i) "
}

PLAB-PLC-v42-NATIVE uses the same version of MyPLC as PLAB-PLC-v42 but uses MyPLC-native rather than the chroot RPM. MyPLC-native is installed on top of a Fedora8 distribution rather than in a separate vserver.

PLAB-PLC-v40 supports PlanetLab 4.0 (it is currently built using MyPLC 0.5.4-2007.03.14 with a couple patches, but with node kernel version 2.6.12-1-F398.7.planetlab from PlanetLab 4.0.0-rc2). PLAB-PLC-v3 supports PlanetLab 3.x. See the end of this entry for notes concerning the 3.x image.

The experiment will take around 5 - 10 minutes to swap in and then another 5 - 10 minutes after that before all the PlanetLab nodes are ready. The larger the experiment, the longer it will take all the PlanetLab nodes to become ready, since they all must download bootstrap data from PLC. It is not necessary to wait for all the PlanetLab nodes to become ready before creating slices. The slices will be created on the nodes as they become ready. Once created, slices on the PlanetLab nodes will become ready within approximately 5 minutes. For information on how to speed this up see section 3.4, Creating a slice, in the MyPLC user's Guide. In particular, make sure you add users and nodes to the new slice, or you will be waiting forever for the slice to appear on the nodes!

Once the PLC is setup, all members of the project will have a PLC account. If you normally have root access on the nodes you will receive the 'admin' role. The creator of the experiment (and the user who swapped it in) will receive the 'pi' and 'tech' roles. These accounts receive PlanetLab related emails. All other users in your project will receive the default 'user' role.

You can browse to your private PLC by visiting https://<plc>.<EID>.<PID>.emulab.net. Once logged in, you can use the MyPLC web interface to add slices, modify user account info, etc.

Your username for PLC is your official Emulab email (i.e., username@emulab.net). You should login to the web interface using this email and your normal Emulab password. All of the version 2 ssh public keys in your ~/.ssh/authorized_keys file will be uploaded to your PLC so that you can login to private PlanetLab nodes easily. You may also add additional keys via the private PlanetLab website.

Once you login to your private PlanetLab nodes, you will notice that they are also Emulab nodes. This means that you will be able to login to the nodes independently of PLC or a specific slice. Normal Emulab tools, such as the event system and linktest, should also run correctly. We have attempted to modify the normal Emulab node boot process so that it does not interfere with PlanetLab node fidelity.

You may swapmod your experiment to add or remove private PlanetLab nodes. However, changing the IP of the PLC node is not supported. This is mainly important when communicaction is done over the experimental LAN interfaces. The easiest way to ensure this is to simply put the PLC node first in the lan string proved to make-lan. The swapmod process will also pick up any user account changes in the experiment's project.

Common Problems

For the PLAB-PLC-v40 image, we have occasionally observed that pc3000 nodes will hang during boot of the PlanetLab node kernel, often after you see a message of the following type on the console output:

Allocate Port Service[0000:00:06.0:pcie00]
Allocate Port Service[0000:00:06.0:pcie01]

This seems to happen only after unclean shutdown (i.e., after forcing a power cycle from users.emulab.net with the node_reboot command). If this occurs, rebooting the node should get it out of the bad state.

Upgrading the PLC Image

See MyPLCInstallNotes.

Legacy PLAB-PLC-v3 image notes

This image supports PLC 3.x, and can be used if you need to run or demonstrate legacy code or applications. These notes describe differences in functionality relative to that supported PLC 4.x PLAB-PLC image, described above.

The PLC setup output is currently only sent to the console. A failed setup will not stop a swapin. Once the PLC Setup completed successfully the file /planetlab/emulab/setup_passed will be written.

All control traffic between the PLC node and the private PlanetLab nodes is over the control net. However, you may connect the PlanetLab nodes themselves together via a private LAN as there is basic support for a private network. In particular there is support for bringing up one private interface on each node, but there is no support for adding routes. This effectively means that all the PlanetLab nodes need to be on the same private LAN. Neither of these constraints is checked right now, so be sure they are met.

You should also turn off Linktest since it will fail anyway.

The output of the individual PlanetLab node setup is also sent to the console. A setup log for each node is also uploaded to

http://<plc>/alpina-logs/nodes/

where <plc> is the host name of the PLC node for your experiment

PlanetLab 3.x supported only one public key, so we upload the passphrase-less version 2 public key file, ~/.ssh/id_dsa. You are free the change it to something else.

All users who normally have root access on the nodes will also have root access on the PlanetLab nodes. You need to login as root using one of your ssh keys found in ~/.ssh/authorized_keys. (Unlike with the slices all your public keys are used)

Finally, note that modifying a plab-in-emulab experiment is currently not supported.