Using QEMU with Olive to emulate Juniper Routers

This guide is to help you run Olive (JunOS running on a standard PC not attached to a PFE) inside QEMU to create a virtual Juniper router lab. You should also be able to attach your QEMU processes to dynamips.

Other sites recommend you download a binary called JEMU.exe from a Chinese forum and use it to run your Olive on. First of all it's a Windows only binary. Windows is not the ideal environment for running QEMU (kqemu comes to mind.) Secondly there is no source provided! Like hell i am going to run a binary i got from a Chinese forum. So i went out and searched for an alternative.

Between the time this tutorial was originally written and now VMWare is supported out of the box with newer JunOS images. However it still may be nice to use qemu to run your lab (studying for an older test for example that is stuck on 8.1 or 8.5), so this tutorial has been updated

Updates

Started from new svn trunk (20080910) and applied latest changes from upstream sources. This also gets rid of the PEMU nic cards which didn't work very well. The eepro100 nic cards work perfectly so nothing is needed.

The problem is that JunOS does not like the new BIOS that qemu started using since version 0.12 call SeaBIOS. You will need to use an older pc-bios or a modified SeaBIOS to run Juniper olives.

Getting and Installing QEMU

As of QEMU 0.12.2 multicast fixes are still not added to eepro100.c, but they have been added to the git repository. This is great news and it means that the fixes should probable be in the next stable release of QEMU.

This also means that you just need to download, compile, and install the latest QEMU git master branch to get Olive running. No more patching needed. (However i am working on a new patch to bring back "udp" to natively connect to dynamips.)

Ubuntu

Dependencies

First we need to get your PC installed with the dependencies to run and compile QEMU.

Install build dependencies (your distribution may require you to manually do this. Consult the QEMU documentation for what is required to build QEMU)

sudo apt-get install build-essential kqemu-source
sudo apt-get build-dep qemu

Fetch the QEMU code from GIT into a temporary directory

mkdir ~/src
cd ~/src 
git clone git://git.qemu.org/qemu.git

Compiling and Installing

Now we have our build dependencies out of the way and we have a patched source tree we to compile it an install it. If you are familiar with any GNU package this will all be similar.

First run configure to and set a prefix (default is /usr/local/) and turn on options provided in the patch

./configure --prefix=/usr/local/ \
            --target-list=i386-softmmu \

After the configure script completes successfully Just issue make and make install.

make -j8
make install

Now your patched QEMU is in the prefix you specified above.

OS X

QEMU 14.1 is available from Macports. Once this is installed, instructions for FreeBSD / JunOS from below can be followed.

FreeBSD

Qemu is available from ports in emulators/qemu .

To build it use portinstall emulators/qemu or pkg_add -rv qemu

For further instruction see the official FreeBSD wiki http://wiki.freebsd.org/qemu

Installing FreeBSD as a guest

JunOS is based on FreeBSD and installing FreeBSD is required when setting up an Olive box.

Download FreeBSD 4.11 mini iso from FreeBSD ftp site

Create a new qcow2 disk image to use for our olive hard drive

qemu-img create -f qcow2 olive-base.img  8G

This creats a new image 8 gig in size which seems to be enough to install and run olive just fine

Now you need to start our modified QEMU and boot from the ISO you downloaded. We'll attach a NIC now and tell it to use the internal natted 'user' mode. This will allow us to scp the jinstall image later on.

qemu -m 256 -hda olive-base.img -cdrom 4.11-RELEASE-i386-miniinst.iso -boot d -localtime -net nic,macaddr=00:aa:00:00:01:01,model=i82559er -net user 

You will now see qemu come up in curses mode which doesn't require X.

Skip the kernel configuration and choose the standard installation.

When you get to partitioning, allocate first the whole disk to BSD, and after that create slices like this:

 ad0s1a /               2000M
 ad0s1b	swap            1024M
 ad0s1e	/config         12M
 ad0s1f	/var            rest 

/ has to be a reasonable size or else you'll run out of space on /mnt.

Choose 'Minimal' installation type and skip installing ports.

After the base is installed it will ask you if you want to use DHCP to configure your NIC. Do this now to allow us to scp your jinstall after we reboot.

Except for the DHCP, choose "no" for everything else (Linux compatibility, NFS, FTP, Inetd etc..)

After the installer completes it will reboot. Right after it reboots close or kill your QEMU process or it will boot into the installer cd again.

Installing Olive

Now we are ready to install Olive into our QEMU host. You will need to obtain a jinstall version. Use something less than 8.5 to get it to properly work. (8.5 is based on FreeBSD 6 and not FreeBSD 4.11).

Now we need to boot our guest again but this time boot from the image. Again we will attach one nic to the 'user' proccess.

qemu -m 256 -hda olive-base.img -boot c -localtime -net nic,macaddr=00:aa:00:00:01:01,model=i82559er -net user

After FreeBSD fully boots login as root. Issue a netstat -r | grep default to see what our default gateway (our host machine is). You need SSH running on your host machine for this to work.

After you find out the default route ip address go ahead and scp the image onto your guest.

scp bbennett@10.0.2.2:~/Desktop/jinstall-8.3R1.5-domestic-signed.tgz /var/tmp

The following is a step that was missing from this guide, if the way I did it is inefficient to you, feel free to edit :)

Modify jinstall file

JunOS image after 7.4 version has a binary called checkpic. This binary will fail and the image cannot be installed. Replacing this binary with /usr/bin/true fixes the issue.

cd /var/tmp
mkdir jinst-signed
cd jinst-signed
tar zxvf ../jinstall-8.3R1.5-domestic-signed.tgz

mkdir jinst
cd jinst
tar zxvf ../jinstall-8.3R1.5-domestic.tgz

mkdir pkgtools
cd pkgtools
tar zxvf ../pkgtools.tgz
cd bin
cp /usr/bin/true ./checkpic
cd ..

tar zcvf ../pkgtools.tgz *
cd ..
rm -rf pkgtools

tar zcfv /var/tmp/jinstall-8.3R1.5-domestic-olive.tgz *

If you want you can recalculate MD5 and SHA1 checksums before archiving them back. This is not required as you can just install the non-signed version of the jinstall package. To sign your images run the following commands:

md5 -q  jinstall-9.0R1.10-domestic-signed.tgz > jinstall-9.0R1.10-domestic-signed.tgz.md5
openssl sha1 jinstall-9.0R1.10-domestic-signed.tgz > jinstall-9.0R1.10-domestic-signed.tgz.sha1

just edit *.sha1 file and remove file name and leaving only checksum inside.

Installing Olive

pkg_add -f /var/tmp/jinstall-8.3R1.5-domestic-olive.tgz

After this finishes issue the halt command and kill your QEMU session window.

First Boot

The jinstall above really just installed a bootstrap environment so you need to boot up the guest OS one more time to finish the installation (depending on your version). If you just restarted your guest above you will notice that you will get no output on your screen. This is because a real Juniper router has no VGA out and redirects everything to the serial port. No worries for us since QEMU will redirect the serial port to either stdio or a telnet port.

Start up our guest one more time this time getting rid of the gui and redirecting the console to a telnet port. Please leave the memory at 256 for this process. After this first boot you can get by with a lot less memory, but we need it now.

qemu -m 256 -hda olive-base.img -boot c -localtime -nographic -bios /path/to/bios-nosmbios.bin

Now wait while the bootstrap process completes. The virtual olive will reboot itself automatically and nothing is needed. At the end of this process we will be sitting at a login prompt. Login as root and issue the halt command and kill your QEMU proccess.

Now you have a base olive hard drive image. QEMU allows you to use this as a base for other harddrive issues and only writing the changes to your 'slave' images saving on disk space!

Running your router

Creating a router instance

Now we will create a new hard drive image off of your base image above. Repeat for all your routers you want to emulate

qemu-img create -b olive-base.img -f qcow2 R1.img

Now you can start your router with the following

qemu R1.img -m 96 -nographic -daemonize -serial telnet::2001,server,nowait -localtime \
-net nic,macaddr=00:aa:00:60:00:01,model=i82559er -net socket,listen=:6000 

Telnet to localhost port 2001 to connect to your new router and start configuring!

If you are looking for a simple way to start and stop routers look at my (extremely) simple script here QEMU_Launch_Script

Networking your routers

QEMU Olive to QEMU Olive

socket

Socket mode creates a tcp stream between two qemu instances with one a client and the other a server.

qemu R1.img -m 96 -nographic -daemonize -serial telnet::2001,server,nowait -localtime \
-net nic,vlan=1,macaddr=00:aa:00:60:00:01,model=i82559er -net socket,vlan=1,listen=:6000 

qemu R2.img -m 96 -nographic -daemonize -serial telnet::2002,server,nowait -localtime \
-net nic,vlan=1,macaddr=00:aa:00:60:00:02,model=i82559er -net socket,vlan=1,connect=127.0.0.1:6000

udp

UDP mode is mainly used to connect to dynamips emulated routers but can also be used as a more reliable way to connect two QEMU olives together. UDP mode can result in much lower latency connections as well.

Currently this is not supported as the patch needs to be rewritten

qemu R1.img -m 96 -nographic -daemonize -serial telnet::2001,server,nowait -localtime \
-net nic,vlan=1,macaddr=00:aa:00:60:00:01,model=i82559er -net udp,vlan=1,sport=10000,dport=10001,daddr=127.0.0.1

qemu R2.img -m 96 -nographic -daemonize -serial telnet::2002,server,nowait -localtime \
-net nic,vlan=1,macaddr=00:aa:00:60:00:02,model=i82559er -net udp,vlan=1,sport=10001,dport=10000,daddr=127.0.0.1

QEMU Olive to Real port

tun/tap

One way to connect an interface on your Olive with a real Ethernet port is to use a bridge and the net=tap option. This requires you to have the generic TUN/TAP driver either built-in to your kernel, or available as a module. To check the availability of this module do the following:

% ls -la /dev/net/tun
crw-rw-rw- 1 root root 10, 200 2007-12-13 17:05 /dev/net/tun

If you get no such file or directory, try doing a modprobe tun. It should then appear in the lsmod output.

Let's say you started the emulator with the following:

 qemu R1.img -m 96 -nographic -daemonize -serial telnet::2001,server,nowait -localtime \
 -net nic,vlan=1,macaddr=00:aa:00:60:00:01,model=i82559er -net tap,vlan=1,script=no 

Note the vlan=1 and -net tap options. This basically connects your Olive's fxp0 to a virtual tap interface (usually tap0) on your host system. Start up Qemu, and once the emulator is running, proceed to the next step. You'll need to start up the emulator as root, or change the ownership or permissions on /dev/net/tun.

We're going to need to now bridge the tap0 interface to another Ethernet interface on the host system. To do this, we'll utilize the bridge capabilities of Linux. First, get the bridge utilities:

% sudo apt-get install bridge-utils

Let's assume you want to connect fxp0 on the Olive to eth1 on the host system. Do the following as root:

brctl addbr br0
brctl addif br0 eth1
brctl addif br0 tap0

This creates a bridge device, br0, and binds the two interfaces to it. Right now everything is still down. Let's bring it up...

ifconfig eth1 up
ifconfig tap0 up
ifconfig br0 up

Now the bridge and member interfaces should be up. Don't assign any IP addresses to either of the member interfaces. If you want to, you can assign something to br0 if needed. (br0 is analogous an SVI in the Cisco world) To see the status of the bridge, do the following:

% brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000cf19ce06c       no              eth1
                                                        tap0

Now, assign an IP address to the fxp0 interface on your Olive, and it should be online. You are also free to tcpdump on the bridge or member interfaces, for debugging. All of this can probably be put into a QEMU interface script, so you can remove the script=no option, and make it a little more automatic.

QEMU Olive to dynamips

The lastest patch also added a udp option to allow networking to a dynamips or pemu. Not Supported yet.

qemu Juniper1.img -m 96 -nographic -daemonize -serial telnet::2001,server,nowait -localtime \
-net nic,vlan=1,macaddr=00:aa:00:60:00:01,model=i82559er -net udp,vlan=1,sport=10000,dport=10001,daddr=127.0.0.1

Then in your dynagen .net file

[[router Cisco1]]
fa0/0=NIO_udp:10001:127.0.0.1:10000

Troubleshooting

Watchdog panic immediately after boot

FreeBSD/i386 bootstrap loader, Revision 0.8
(builder@vouivre.juniper.net, Fri Aug 24 16:01:34 GMT 2007)
Loading /boot/defaults/loader.conf
/boot/installer text=0x24c77f data=0x2aa7c+0x38462
syms=[0x4+0x3a420+0x4+0x454ac]
|
Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [installer]...
kernel trap 12 with interrupts disabled
instruction pointer     = 0x8:0xc02ebc3e
stack pointer           = 0x10:0xc07f9edc
frame pointer           = 0x10:0xc07f9f34
code segment            = base 0x0, limit 0xfffff, type 0x1b
                       = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, IOPL = 0
current process         = Idle
interrupt mask          = net tty bio cam
trap number             = 30
dog: ERROR - reset of uninitialized watchdog
panic: unknown/reserved trap
(null)(c037dbc0,c037dbc0,c032d5c4,c07f9df0,5) at0
(null)(c032d5c4,1e,c07f9f34,0,0) at0
(null)(c07f9e9c,0,c07f9ef4,c01951ed) at0
(null)(10,10,10,0,c) at0
(null)(10,10,10,0,7fe000) at0
(null)(2) at0
(null)(c07f9ff4,c02e569c,0,81,7fe000) at0
(null)(c07f9fd4,f,3,8,0) at0
(null)(7fe000,0,0,0,0) at0
(null)() at0
dog: ERROR - reset of uninitialized watchdog
dog: ERROR - reset of uninitialized watchdog
Uptime: 0s

If you receive this error immediately after trying to boot your Olive for the first time it means you are running Qemu 0.12.0 or later with the SeaBIOS. This bios currently has a bug that causes JunOS kernel to hang. You can use a modified version of the bios as a workaround to this issue.

You can download a already compiled version of this bios here.

Alternatively you can compile your own modified version of the SeaBIOS:

First download the lastest copy of the source code, then revert the git repo to a version confirmed to work.

$ git clone git://git.linuxtogo.org/home/kevin/seabios.git seabios
$ cd seabios
$ git checkout 8c8a880b584ccf8958d67e99a6750ba32d0b6454

Edit the src/config.h file and set the CONFIG_SMBIOS to 0 like below

// Support generation of SM BIOS tables (for emulators)
#define CONFIG_SMBIOS 0 

Build seabios:

$ make

Copy the bios.bin somewhere on your file system. I like to name this bios-nosmbios.bin and put it in the $PREFIX/share/qemu directory with the non-modifed bios.bin file.

cp out/bios.bin /usr/local/share/qemu/bios-nosmbios.bin