Flying Linux Containers on Xen 4.0.* on PV Ubuntu 10.04 Lucid

Do you have 3 minutes and 17 seconds of your time to spend, well, and some download time?

I was dreaming for some time now about a possibility to run virtualized machines inside one PV xen instance in a cloud like Amazon or similar… That would be great, but several references on the net state that ex. amazons cloud can not – will not run dom0 kernels. Lately I found a reference stating that it became possible to run own kernels like ones having container capabilities…, well not dom0 but… It would certainly be exciting to see Linux Containers “lightest” virtualized birds on “heaviest” Xen, don’t you think so? (“heaviest” == coolest đŸ˜‰ )
Unfortunately making that work on my test Xen PV instance proved not to be so easy…

  • PV Ubuntu 10.04 Lucid with stock kernel linux-image-2.6.32-24-generic-pae (didn’t know that will also work as PV) running on
  • Ubuntu 9.10 Karmic xen-hypervisor-4.0 4.0.1rc5-0ubuntu1 and dom0 kernel linux-image- v. 100803222122.
  • Latter tested on a real system without Xen and confirmed to work just as well.

As you can see at the moment you can find far more complaints about (im)maturity and usability of Linux Containers (LXC) than working examples. Actually none of them really worked for me. Closest came lxc-ubuntu script (Copyright (C) 2010 Nigel McNie – Modified for Ubuntu 2010 phbaer), but even that didn’t not work for me for interactive ssh sessions and console, only scp. I took more than two weeks of my time to get into intricacies of LXC and it wasn’t an easy task, but I think the result is worth it. As I couldn’t find working lxc ubuntu template anywhere I patched the one from latest ubuntu lxc testing sources to make it work and a bit more Ubuntu like (most notable first user in admin group and no root logins).

I added one script to create networking environment on hosting PV instance that could be used remotely in the cloud while not messing up with the working default gateway interface as we might loose connectivity.

# export LXCBASE=.../  # default demo base root dir
LXCBASE=${LXCBASE:-/lxc}; sudo mkdir -p $LXCBASE
wget -q -O - | \
catdoc >; sh ./; rm; sudo tar xzvf Ubuntu1004LXCflight.tgz -C $LXCBASE

The following does quite some weight lifting so make sure it doesn’t brake something in your environment before you run it. You can copy paste it line by line to test or run the whole script ./runscript

# runscript
# (C) 2010 Virtual User <>, GPLv2

# export LXCBASE=.../
LXCBASE=${LXCBASE:-/lxc}  # default demo base dir

#cd to lxc demo directory

### Cleanup of previous demo attempt / environment
# purge previous myUbuntu demo container
lxc-stop -n myUbuntu; lxc-destroy -n myUbuntu
rm -rf myUbuntu
# Purge previous cache of lxc-ubuntu script
#rm -r /var/cache/lxc/ubuntu/
# Remove previous bridge br0
ifconfig br0 down; brctl delbr br0
# make backup of current dhcp3 & bind9
tar czvf lxc-demo-backup-$(date +%y%m%d%H%M).tgz /etc/bind/ /etc/dhcp3/ /etc/default/dhcp3-server
# Purge lxc demo related software
apt-get -y purge lxc dhcp3-server bind9

# Demo START
# Make localized config file
head -n15 ./ | \
        sed "s/\bserver1\b/$(hostname)/" > $(hostname)
# Create lxc environment
apt-get -y install lxc dhcp3-server bind9 expect tcllib
./ # Also installs lxc dhcp3-server bind9
# Local test config for dhcpd
cat << EOF | tee -a /etc/dhcp3/dhcpd.conf 

subnet netmask {
  option domain-name-servers;
  option domain-name "lan";
  option routers;
  option broadcast-address;
  default-lease-time 600;
  max-lease-time 7200;

host myUbuntu{
  hardware ethernet 86:d7:66:8a:04:90;
sed -i 's/INTERFACES=""/INTERFACES="br0"/' /etc/default/dhcp3-server
/etc/init.d/dhcp3-server restart

# Create an image of a new container with working Ubuntu Lucid
firstuser=me firstpass=mypassword ./lxc-ubuntu -p /$LXCBASE/myUbuntu

# Add MAC hwaddr to config to keep the same IP on container restarts
sed -i "/^ =/ a\\ = 86:d7:66:8a:04:90" myUbuntu/config

# Copy X install script into home directory of the admin user in the new container.
cp -va myUbuntu/rootfs/home/me/
# Set vncpasswod of the admin user
expect_run() { cmd64=$(echo -n $1 | base64); pas64=$(echo -n $2 | base64)
expect -c 'package require base64
set cmd [base64::decode "'$cmd64'"]
set pas [base64::decode "'$pas64'"]
if 1 [linsert $cmd 0 spawn ]
while true {
  expect -re "Password:$" { send "$pas\r"} \
  expect -re "Verify:$" { send "$pas\r"} \
  expect -re "s password: $" { send "$pas\r"} \
  expect -re "password for *: $" { send "$pas\r"} \
  eof { break}
}'; }
mkdir -p myUbuntu/rootfs/home/me/.vnc
expect_run "vncpasswd myUbuntu/rootfs/home/me/.vnc/passwd" mypassword
chown -Rv 1000:1000 myUbuntu/rootfs/home/me/.vnc

# Expand var/cache/apt/archives saved by
# tar cf aptcache.tar myUbuntu/rootfs/var/cache/apt/archives/*.deb
[ -r ./aptcache.tar ] && tar xf ./aptcache.tar

# Create & start LXC container
lxc-create -n myUbuntu -f /$LXCBASE/myUbuntu/config
lxc-start -n myUbuntu -d; sleep 15

# ssh into it, install very basic X workstation & start it
expect_run "ssh -t me@ ./" mypassword
expect_run "ssh me@ /usr/bin/vncserver -localhost" mypassword

# Connecting instructions
EXTERNAL_INTERFACE=$(ip route | sed -n 's/  *metric.*//;s/default via [0-9.]* dev //p')
EXTERNAL_IP=$(ip addr show dev $EXTERNAL_INTERFACE | sed -n 's|/.*||;s/ *inet //p')
cat << EOF
### Contact internal container by NAT through external interface $EXTERNAL_INTERFACE
### from anywhere by
sudo apt-get -y install  xvnc4viewer openssh-client
ssh -N -f -p 2220 me@$EXTERNAL_IP -L 5911:
SSHPID=\$(ps ax | grep -v grep | \\
  grep "ssh -N -f -p 2220 me@$EXTERNAL_IP -L 5911:" | sed 's/^ *//;s/ .*//')
vncviewer localhost:11  # port 5911 tunneled above

### Remeber to kill ssh tunelling client running in background when
### you are finished using the vncviewer by:
kill \$SSHPID

I wonder how long does it take? (all packages already downloaded to cache)

time ./runscript
# . . .
### Contact internal container by NAT through external interface eth0
### from anywhere by
sudo apt-get -y install  xvnc4viewer openssh-client
ssh -N -f -p 2220 me@ -L 5911:
SSHPID=$(ps ax | grep -v grep | \
  grep "ssh -N -f -p 2220 me@ -L 5911:" | sed 's/^ *//;s/ .*//')
vncviewer localhost:11  # port 5911 tunneled above:

### Remeber to kill ssh tunelling client running in background when
### you are finished using the vncviewer by:
kill $SSHPID

real    3m16.794s
user    0m4.816s
sys     0m8.473s

Let me know if anyone makes it actually fly in the Amazons cloud. Any suggestions how to test this on some free cloud infrastructure?

3m16.794s … Was this one perfect flight or what?Enjoy your flying

Contrat Creative Commons
This entry was posted in LXC, Xen. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s