Installing Emulab Software on boss
This procedure is similar to installing the testbed software on ops. Before beginning, make sure that ops is installed, running, and reachable through the network by your new boss machine.
Step 1 - Installing Packages
Unpack the Emulab Server Package
someplace. Let's say you unpacked it to /usr/tmp. You would
then run:
ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl env PKG_PATH=/usr/tmp/packages pkg_add emulab-boss-2.1
Of course, if you unpacked it somewhere else, put in the correct path.
We provide all of the binary packages necessary to build boss in the packages
tarball. If you would like to install additional software on boss, unpack the
'approved' ports collection to a place
such as /usr/ports (rename or delete the old directory, if it
exists). The package unpacks to approximately 175MB. Install whichever software
you would like.
Step 1a - A fixup
A number of sites are having problems related to a broken perl
installation. Briefly, the installed perl is not built with the
SETUID option, which causes the Emulab installation
to fail in odd ways. Lets check:
pkg_info -L perl-5.8.8_1.tbz | grep sperl
Make sure that it shows /usr/local/bin/sperl5.8.8
in the output. If not, then:
pkg_delete -f perl-5.8.8_1 env PKG_PATH=/usr/tmp/packages pkg_add perl-5.8.8_1.tbz
Step 2 - Unpacking and running 'configure'
Unpack the Emulab source tree to /usr/testbed/src/testbed. Then
run its configure script, providing the local path of your definitions file:
mkdir -p /usr/testbed/src/testbed mkdir -p /usr/testbed/obj/testbed cd /usr/testbed/src/testbed tar ... cd /usr/testbed/obj/testbed /usr/testbed/src/testbed/configure --with-TBDEFS=/path/to/your/defs-file
Remember, you have to use the same defs-file that you used on ops.
There is a FreeBSD-6.3-specific bug in the named configuration files. We have
provided a patch which should now be located in
/usr/testbed/src/testbed/named/rc.named.6.3.patch. You must
apply this patch if you are running FreeBSD 6.3. To do so, run the
following command:
patch -p0 < /usr/testbed/src/testbed/named/rc.named.6.3.patch
Step 3 - Running the boss installation script
In the object tree you've configured (say, /usr/testbed/obj/testbed),
there's an 'install' subdirectory with a script called 'boss-install'. Run
this script as root (not the same package directory argument as above):
cd install perl boss-install -p /usr/tmp/packages
Like the ops-install script, boss-install sets up passwordless sudo for anyone in the wheel group. If this script fails, send the output to Utah so we can fix it. Note that boss-install requires internet access so it can grab some files.
The source installation is performed by the boss-install script and should be completely automatic. However, you may check it or fix up problems by running the makes yourself, as follows:
gmake gmake boss-install gmake post-install
The post-install target needs to be run as root, so that it can make certain scripts setuid, etc.
Step 4 - Verify SSH configuration
The boss-install script should have set up root ssh from boss to ops. To confirm this, make sure this works:
boss> sudo ssh ops ls /
If this fails, you will need to set this up by hand, typing a password:
scp /root/.ssh/identity.pub ops:/root/.ssh/authorized_keys
Step 5 - Setting up named
The testbed software manipulates DNS zone files for two reasons. First, it adds your nodes to them so that you don't have to. Second, it creates CNAMEs for all the nodes in every experiment (so that you can use, for example, 'nodeA.myexp.myproj.emulab.net' to refer to your node regardless of which physical node it got mapped to).
The named_setup script does this by generating zone files. In general, it
concatenates a '.head' file, written by you, with its own generated entries.
The main zone file is /etc/namedb/OURDOMAIN.db, where
OURDOMAIN is from your defs file. (OURDOMAIN, unless explicitly specified,
is taken to be the domain portion of BOSSNODE). We also generate reverse
zone files (for inverse lookups, i.e. turning IP addresses back into names)
in /etc/named/reverse/SUBNET.db, where SUBNET is the class-C
subnet in which the addresses reside (i.e. 10.0.0.db). This value is
defined in the defs file, as is TESTBED_NETWORK.
boss-install makes a reasonable attempt to create a set of named configuration
files for you, placing them in /etc/namedb. If your testbed
consists of a single class-C network, then these files will most likely be
correct, although you'll want to look at them to make sure. Look at these
files to make sure:
/etc/named/OURDOMAIN.db.head /etc/named/reverse/SUBNET.db.head /etc/named/named.conf
If you have more than one class-C subnet for your testbed, you'll need a copy
of the reverse zone file for each one. You want to put boss, ops, and any
'infrastructure' equipment such as routers and switches into the zone files.
These zone files do not need to include the nodes - they will be added
automatically. Be sure to edit /etc/named/namedb.conf if you
add any reverse map files (follow the format for the existing entry).
Once you think you've got things set up, run make sure the following command doesn't give you any error messages:
/usr/testbed/sbin/named_setup
This command will generate the following files:
/etc/namedb/OURDOMAIN.db /etc/namedb/reverse/SUBNET.db
*** If you are using unroutable private IP addresses for part of the testbed: ***
In order to handle this situation, we'll need to use a feature of bind called 'views' so that the private addresses are only exposed to clients within the testbed. See the bind documentation for more details on this feature. You'll want to make sure that loopback queries from boss itself see the internal view - you want boss to resolve its own hostname to its private address, not its public one.
In order to use multiple views, we generate multiple zone files. In addition to OURDOMAIN.db, which will be used for the 'external' view, we create OURDOMAIN.internal.db for use with the 'internal' view. So, you'll also need to create OURDOMAIN.internal.db.head. When we generate the zone files, only publically-routable addresses are put into OURDOMAIN.db. OURDOMAIN.internal.db contains all addresses, both public and private. So basically, you'll want to put the public addresses for boss, ops, etc. into OURDOMAIN.db.head, and their private addresses into OURDOMAIN.internal.db.head.
Step 6 - Miscellaneous configuration
There are a few things that we haven't been able to completely automate just yet, though we hope to soon.
hosts file - You want to put boss/ops name/IP addresses in
/etc/hosts on both boss and ops to avoid boottime circular
dependencies (because of NFS cross mounts). This is done for you in ops-install
and boss-install, but you should confirm that it was done properly. If you
change the IP addresses of boss/ops later, you will want to be sure to update
/etc/hosts on both machines.
SSL Certificates - Our apache config file expects to find SSL certificates in:
/usr/local/etc/apache/ssl.crt/www.<sitename>.crt /usr/local/etc/apache/ssl.key/www.<sitename>.key
Where <sitename> is OURDOMAIN from the configure defs file, which defaults to boss' domain name.
boss-install already generated a temporary no-passphrase certificate for you and placed it in the locations specified above. However, we recommend that you get a "real" certificate from Verisign (or one of the others).
DHCPD - boss-install generated a dhcpd.conf.template file and installed
it in /usr/local/etc (the template is derived from information
you provided in the defs file). It then generated an actual dhcpd.conf file
and started up dhcpd for you. Do not edit the dhcpd.conf file directly! If
you need to make changes, change the template instead and then run:
/usr/testbed/sbin/dhcpd_makeconf -i -r
locate database - It can be useful to update the 'locate' database to help you find files as you're learning the system. This normally happens nightly, but you can force it to happen now by running the following as root:
/usr/libexec/locate.updatedb &
This will run in the background for several minutes. You can then find 'foo.conf' by running 'locate foo.conf'.
dhclient - If you don't control the public network to which boss and ops
(and perhaps tipserv) are attached, your network admins may want you to run
dhclient on those interfaces (and you may want to as well so that your boss
/etc/resolv.conf contains the correct nameserver returned by
dhclient, to avoid configuring it statically). However, boss runs its own
nameserver that ops and the experimental nodes must contact to resolve
internal names, so you must get dhclient to correctly write
/etc/resolv.conf (i.e., boss must be the first nameserver). The
easiest way to solve this problem is to add the following line to boss'
/etc/dhclient.conf file:
prepend domain-name-servers 127.0.0.1
and add the following line to ops' (and any other non-experimental nodes, such
as tipservs) /etc/dhclient.conf:
prepend domain-name-server <bossip>
where <bossip> is replaced by a valid IP address for your boss. Make
sure to restart dhclient on these machines since dhclient will continue to
overwrite /etc/resolv.conf incorrectly.
Step 7 - Reboot boss
Okay, go ahead and reboot boss now, and make sure it comes up okay.
