Playing with (an) Orchestra

I have recently been working on a project using Orchestra.  Orchestra is a great provisioning server for automatically deploying Ubuntu machines on hundreds of servers.  I wanted to play with it a bit before diving in but I didn't have any "bare metal" handy.  This is my virtualized test setup for experiments using my laptop.

Internet (via the vm host)  <-->  Orchestra Server (vm guest)  <-->  Client machines (vm guests)

Build the Orchestra server
First get the ubuntu oneiric iso for our Orchestra server:

$ wget http://releases.ubuntu.com/oneiric/ubuntu-11.10-beta2-server-i386.iso

Next, install the packages we need on the laptop:

$ sudo apt-get install qemu-kvm kvm-pxe

Now build a virtual disk image:

$ qemu-img create -f qcow2 orchestra.img 10G

Launch a virtual machine to install the orchestra server:

$ qemu -m 2047 -hda orchestra.img -net nic,vlan=0 -net user,vlan=0 -redir tcp:5022::22 -redir tcp:5080::80 -net nic,vlan=1 -net socket,vlan=1,mcast=230.0.0.1:1234 -net dump,vlan=1,file=capture.pcap -cdrom ubuntu-11.10-beta2-server-i386.iso

This boots up a machine with two network cards.  The first (eth0) will use your laptop's network connection to give you a connection to the internet.  The second (eth1) is connected to the virtual switch and will be used to talk to fresh machines that need to be provisioned.

Note, we are also using QEMU's "-redir" command to port forward from our host machine into the vm instance. With the configuration above, host ports 5022 and 5080 are redirected to the orchestra server vm ports 22 and 80 respectively. This will allow us to use ssh and http from our host.

Install the ubuntu server as normal.  Select eth0 as your primary network card.

Networking
We're going to set up the orchestra server to also act as our internet gateway for newly provisioned machines.  Please note that the network described below is not secure and should not be used in a production deployment.  Once the server has booted, set up the networking as follows:

/etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet static
    address 192.168.16.1
    netmask 255.255.255.0

Uncomment the following line in /etc/ufw/sysctl.conf

net/ipv4/ip_forward=1

Change the default FORWARD and INPUT firewall rule to ACCEPT in /etc/default/ufw

DEFAULT_FORWARD_POLICY="ACCEPT"
DEFAULT_INPUT_POLICY="ACCEPT"

And add the following lines to the TOP of /etc/ufw/before.rules

# nat Table rules
*nat
:POSTROUTING ACCEPT [0:0]

# Forward traffic from eth1 through eth0.

-A POSTROUTING -s 192.168.16.0/24 -o eth0 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT

Now enable the firewall:

$ sudo ufw disable && sudo ufw enable

Reboot the VM to make sure the networking configuration takes effect

$ sudo shutdown -r now

Installing Orchestra
Once the networking is configured, update it and install Orchestra:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install ubuntu-orchestra-server

Use the following settings:

Password for cobbler user:	cobbler (or any other password, just dont't forget!)
Boot and pxe server IP address:	192.168.16.1
Enable Orchestra managed DNS/DHCP:	yes
Network range for DHCP clients:	192.168.16.100,192.168.16.200
Default gateway for dhcp clients:	192.168.16.1
Domain name for DHCP clients:	<leave blank>

Now sync the cobbler server to activate the changes:

$ sudo cobbler sync

First PXE Boot
Now that Orchestra is up and running, let's get right to the good stuff and PXE boot a new VM.

We need a new disk image:

$ qemu-img create -f qcow2 client.img 10G

Now we just need to start it up:

$ qemu -hda client.img -net nic -net socket,mcast=230.0.0.1:1234 -boot once=nc

The "-boot once=nc" tells qemu to try booting off the network first ("n"), then off the hard disk ("c").

You should get a nice menu on your client VM.  Scroll down to "oneiric-i386-ju ju" and hit enter.  Watch as your new machine is automatically installed!

While you wait for that, have a poke around the web interface for cobbler:

Web Interface
On your host machine, point your browser to:

http://localhost:5080/cobbler_web/

The username is cobbler and the password is cobbler (unless you set a different password during the install)

XMLRPC API
Here's a little snippet of python to provision a server via the API (See https://fedorahosted.org/cobbler/wiki/CobblerXmlrpc for full docs)

import xmlrpclib

server = xmlrpclib.Server("http://localhost:5080/cobbler_api")
token = server.login("cobbler","cobbler")
system_id = server.new_system(token)

server.modify_system(system_id, "name","new-machine",token)
server.modify_system(system_id, "hostname","new-machine.example.com",token)
server.modify_system(system_id, "modify_interface", {
    "macaddress-eth0"   : "10:20:30:40:50:60",
    "ipaddress-eth0"    : "192.168.16.20",
    "dnsname-eth0"      : "new-machine.example.com",
    }, token)
server.modify_system(system_id,"profile","lucid-i386-juju",token)

server.save_system(system_id, token)
server.sync(token)

Now start up a new instance with the MAC address we used above:

$ qemu-img create -f qcow2 client-lucid.img 10G
$ qemu -hda client-lucid.img -net nic,macaddr=10:20:30:40:50:60 -net socket,mcast=230.0.0.1:1234 -boot once=nc

Enjoy!

QEMU Networking

I've been playing with PXE booting and I needed a test setup that I could run from my laptop.  I wanted to create a small isolated network between a few virtual machines.  I generally use virtualbox but for this project I decided to QEMU.

I had just a few requirements:

• I wanted to record the traffic to pcap files for wireshark analysis.  This is essential for diagnosing PXE failures and for understanding exactly what's going on.
• I needed to add and remove machines easily without changing the configuration on the other virtual machines.
• I wanted it to be easy.  No root access, no messing with bridging or tun/tap interfaces.

QEMU Networking

QEMU has a few handy features which make this really easy.  For the full story check out the QEMU Networking page that Mark McLoughlin put together.

Multicast networks allow multiple VMs to communicate with each other just like if they were all connected to a single hub.  Any ethernet frame sent to the network interface on one machine gets sent to all other machines.

-net socket,mcast=239.255.0.1:1234

The second feature is the "dump" network type which will dump any packet to a file.

-net dump,file=log.pcap

Because the multicast "hub" sends all frames to all virtual machines you only need to use the "-net dump" on one of the machines and you will capture all packets.

Launching the VMs

Launch the first machine like this:

$ qemu -hda one.img -net nic -net socket,mcast=239.255.0.1:1234 -net dump,file=log.pcap

And all other machines like this:

$ qemu -hda two.img -net nic -net socket,mcast=239.255.0.1:1234

Multiple NICs

If you have more than one network interface on a machine, you must use the "vlan" option to make sure the options are applied to the correct interface.

If I wanted a gateway VM with eth0 connected through the host to the internet, and eth1 connected to all other virtual machines, it would be launched like this:

$ qemu -hda one.img -net nic,vlan=0 -net user,vlan=0 -net nic,vlan=1 -net socket,vlan=1,mcast=239.255.0.1:1234 -net dump,vlan=1,file=log.pcap

Live capture in Wireshark

If you'd like to see the packet capture in realtime you can use mkfifo to create a FIFO to stream the packets into wireshark's live capture display.  Set it up like this:

$ mkfio live.pcap
$ wireshark -k -i live.pcap &
$ qemu -hda one.img -net nic -net socket,mcast=239.255.0.1:1234 -net dump,file=live.pcap

Enjoy!