Soekris / FreeBSD / Wireless router / DiskLess

Using FreeBSD and a Soekris board.
Last changed: 2002-01-12. © Dirk-Willem van Gulik under this license.

Thanks:
Marten Vijn, Cliff Skolnick for hints and debugging.

Various configurations are possible: PCI or PCCard based. The board is low power enough to be fed by a solar panel and a motercycle overnight 4 amph battery.

Note:If you find any faults, no matter how small - PLEASE email me: dirkx-at-webweaving.org. Thanks!

Below should get you started. This methods lets you use PXE booting/diskless booting first - and once you are 100 satisfied with the configuration to go to a CF card. This means that you can simply powercycle the soekris at will and can experiment with a wide range of settings without being restrained by CF card size, limited writablility and so on.

  1. Requirements

    • Soekris box; type does not matter and powersupply.
    • Serial cross over cable.
    • FreeBSD machine and a functional home network.
    • Ethernet cable from Soekris to the home network
    • Full FreeBSD sources and a 1Gb free.
    • About an afternoon of time.

  2. Initial COM setup

    Connect the soerkis unit to your serial port using a cross over cable. See Stokely's website on Serial Cables for details. Any medium sized computer ship will have them in stock. Make sure both ends are female. Belkin F3B207-06is fine; both www.vikingdirect.nl and zero6.nl have those. But any other cross over type will do.

    Next edit your /etc/remote file and add an entry like:

     
    	com0:dv=/dev/cuaa0:br#19200:pa=none:
    	com1:dv=/dev/cuaa1:br#19200:pa=none: 
    

    Next make sure that a user can write to these /dev/cuaa# devices.

    Typically these are

     
    	crw-rw---- 1 uucp dialer 28, 128 Sep 2 14:35 /dev/cuaa0 
    

    (or they can be made so with chmod g+rw /dev/cuaa0 and chgrp dialer /dev/cuaa0). Meaning that anyone belonging to the group dialer may work with the device. If you do not want to work as root add a user to the dialer group by editing the /etc/group file. Modify the dialer line in that file:
     
    	dialer:*:68:wheel,peter,jon,mary 
    

    Log out/in and verify with the command id that you belong to thar group:

     
    	$ id 
    	uid=1002(mary) gid=1002(mary) groups=666(admins), 68(dialer) 
    

    Now issue the command:

     
    	tip com0 
    

    You should see 'Connected'.

    Next power up the device. You should see garbage or ascii scrolling. If not; try the other com port.

    Note that to exit 'tip' one types '~.' or 'return', '~.'. If you are in ssh or telnet from a remote location you may find that the '~' is eaten by your connection deamon - it is also the escape character for the latter. You can circumvent this by then typing '~~.' -or- by using the -e flag:
    	ssh -e none host...
    

  3. The next step is to make sure you 'agree' on a baud rate. If the above gave you readable text - you are in luck. Otherwise edit the /etc/remote file and change the 19200 to 9600. This should give you readable text.

    If not - see the soekris site for other possible baudrates.

    When it all works you should see this after powerup:

    comBIOS ver. 1.05  20020419  Copyright (C) 2000-2001 Soekris Engineering.
    
    Soekris Engineering net4501               CPU 80486 134 Mhz 
    
    0064 Mbyte Memory
    
    
    PXE-M00: BootManage UNDI, PXE-2.0 (build 082)
    
    Slot Vend Dev  ClassRev Cmd  Stat CL LT HT  Base1    Base2   Int 
    -----------------------------------------------------------------
    0:00 1022 3000 06000000 0006 2280 00 00 00 00000000 00000000 00
    0:16 13A3 0012 0B400001 0116 0280 10 3C 00 A0000000 A0001000 10
    0:17 104C AC51 06070000 0107 0210 10 3F 82 A0002000 020000A0 11
    0:18 100B 0020 02000000 0107 0290 00 3F 00 0000E101 A0004000 05
    0:19 100B 0020 02000000 0107 0290 00 3F 00 0000E201 A0005000 09
    
     5 Seconds to automatic boot.   Press Ctrl-P for entering Monitor.
    
  4. Now configure the device to boot using PXE. This is the default. If changed use 'ctrl-P' to get into the BIOS just after power on. When done correctly you should see:
    BootManage UNDI, PXE-2.0 (build 082)
    BootManage PXE-2.0 PROM 1.0, NATSEC 1.0, SDK 3.0/082 (OEM52)
    Copyright (C) 1989,2000 bootix Technology GmbH, D-41466 Neuss.
    PXE Software Copyright (C) 1997, 1998, 1999, 2000 Intel Corporation.
    Licensed to National Semiconductor
    
    CLIENT MAC ADDR: 00 00 24 C0 3C AC  
    
  5. DHCP Server

    Make sure that the DHCP deamon: dhcpd is installed. If not install it from CD or over the net from a precompiled package: it as a package:

    	pkg-add -r isc-dhcp3
    
    or build from ports:
    	cd /usr/ports/net
    	cd isc-dhcp3
    	make && make install
    
  6. Now edit the /usr/local/etc/dhpcd.conf:
    # dhcpd.conf
    #
    option domain-name "YOURDOMAIN.COM";
    option domain-name-servers YOURDNSSERVERIP;
    option routers YOURROUTERIP;
    ddns-update-style none;
    default-lease-time 600;
    max-lease-time 7200;
    authoritative;
    
    log-facility local7;
    subnet 10.11.0.0 netmask 255.255.255.0 {
      range 10.11.0.200 10.11.0.210;
    }
    server-name "YOURSERVERNAME";
    server-identifier YOURSERVERIP;
    next-server YOURSERVERIP;
    
    host sjoesjoe {
      hardware ethernet  00:00:24:c0:39:9c;
      fixed-address FQDN;
      option root-path "YOURSERVERIP:/mboot";
      option host-name "FQDN";
      filename "pxeboot";
    }
    
    If your DNS is not fully set up; use an IP address rather than a FQDN.

  7. Either change the installed /usr/local/etc/rc.d/isc-dhcpd.sh.sample to be ran at boot time;
    	mv /usr/local/etc/rc.d/isc-dhcpd.sh.sample \
    		 /usr/local/etc/rc.d/isc-dhcpd.sh
    

    and/or run it now manually:

    	/usr/local/etc/rc.d/isc-dhcpd.sh.sample start
    
  8. Trying the DHCP server.

    Restart the DHCP server (killall dhcpd; dhcpd; check the error log tail /var/log/messages) and then power cycle the soerkis.

    You should now see:

    	CLIENT MAC ADDR: 00 00 24 C0 3C AC  
    	CLIENT IP: 10.11.0.105  MASK: 255.255.255.0  DHCP IP: 10.11.0.2                
    	GATEWAY IP: 10.11.0.1
    

    If not you may try tcpdump ether host 00:00:24:C0:3C:AC (substitute for your mac adddress) to see what is avry.

  9. Setting up TFTP

    The next step is to set up tftp to allow the soekris to get the pxeboot file specified above.

    If you are using etherboot rather than the standard PXE boot - skip this step. Etherboot uses NFS for sourcing the kernel.

    Also be aware of the LOADER_TFTP_SUPPORT=YES default in /etc/make.conf which can be disabled to simply the boot procedure.

    Create a directory like /tftpboot and copy pxeboot into it.

    	mkdir /tftpboot
    	cp /mboot/boot/pxeboot /tftpboot
    	chmod a+rx /tftpboot
    	cmod a+r /tftpboot/pxeboot
    

    Next edit /etc/inetd.conf and add:

    tftp    dgram   udp     wait    nobody  /usr/libexec/tftpd      tftpd -l /tftpboot
    

    Then restart inetd.conf (with killall -1 `cat /var/run/inetd.conf`). To make sure that inetd is started at boot time; ensure that in your /etc/rc.conf you have the entry:

    	inetd_enable="YES"
    
  10. Testing the PXE booting

    Powercycle the soekris again and verify that you see:

    	PXE Loader 1.00                                                                
    
    	Building the boot loader arguments
    	Relocating the loader and the BTX
    	Starting the BTX loader
    	Console: internal video/keyboard
    
    	PXE version 2.1, real mode entry point @9e79:0106
    	BIOS 577kB/64512kB available memory
    

    If not; check that /tftpboot/pxeboot exist, that you can 'tftp' to the machine; that inetd was restarted and/or use the above tcpdump to see what is going on. The -l argument in the /etc/inetd.conf line ensures that any requests are logged in /var/log/messages.

  11. Preparing a bootable section.

    You have several options. Including simply using the one of the freebsd server. The method described below will help you build a totally separate tree - so you can edit/modify in place.

    It may be advisable to unset the flag TERM_EMU. Either by editing /sys/boot/i386/libi386/Makefile or by undef-ing TERM_EMU in the make. This dums down the terminal emulation to a pure serial ascii link; with no VT100 or other escape characters to get things in fancy bolds or other colours.

    Using make world DESTDIR=/mboot it is possible to create an entirely separate tree. See jail(8) for more information. The general procedure (which you only need to do once is):

    	cd /usr/src
    	mkdir -p /mboot
    	make world DESTDIR=/mboot
    	cd /usr/src/etc
    	make distribution DESTDIR=/mboot -DNO_MAKEDEV_RUN
    	cd /mboot/dev
    	sh MAKEDEV
    

    This will create you an entire environment (minus the kernel). Consult the /usr/src/Makefile* and make.conf for some extra options and methods for not building/including certain sections; such as man pages and games. Within this tree you can freely delete/rename things to pair things down (see the end of this story for a better method based on PICOBSD).

    Note: You are going to need some 150-200Mb for this 'full' FreeBSD install (at least initially - once you are done experimenting it can be paired down to just a few meg. So you may want to make the above /mboot a symlink to a location with a lot of space; or substitute /mboot for /usr/local/somewhere in the rest of this document.
  12. The next step is to build a kernel.

    One can use the standard GENERIC kernel - but it is strongly advisable to use a stripped down kernel. A sample is here. Save this file in /sys/i386/conf

    The resulting kernel is then copied into the boot section.

    	cd /sys/i386/conf
    	fetch http://www.webweaving.org/wlg/SJOESJOE
    	config SJOESJOE
    	make
    	cp kernel /mboot/boot/kernel/kernel/
    

    Alternatively you can of course do a make install with a DESTDIR pointing to your /mboot: make install DESTDIR=/mboot

  13. Setting up NFS
  14. Ensuring that the serial port is the console:

    Normally if no vga is present; the first serial port will be the console. This can be overriden by valuesin the BIOS, in the bootloader, the dhcp information, the pxe second stage loader; by the loader.rc file in /boot, the device.hints file and by values statically compiled in the kernel.

    For more details see the section in the handbook on the serial console or the quick instructions below.

    On 4.x and before; you want to set bit 6 of the sio0 serial device. See also the manpage of sio(4).

    	device    sio0    at isa? port IO_COM1 flags 0x30 irq 4
    
    	# if you have silo overflows/problems with speed/FIFO
    	# device    sio0    at isa? port IO_COM1 flags 0x32 irq 4
    
    If you are using 5.x or newer you may have to set the sio.0 flag in /mboot/boot/device.hints:
    	#..../boot/device.hints
    	hint.sio.0.at="isa"
    	hint.sio.0.port="0x3F8"
    	hint.sio.0.flags="0x30"
    	hint.sio.0.irq="4"
    	hint.sio.1.at="isa"
    
    or, if you had a hints directive:
    	# Kernel config file SJOESJOE
    	hints "generics.hints"
    
    in your kernel, in the file referenced there (generics.hints). Note that the serial section in the handbook also covers this in detail. And again; check the section in the handbook on the serial console specifically.

  15. Configuring the diskless system:

    Ensure that your /etc/rc.conf contains

    	nfs_server_enable="YES"
    	portmap_enable="YES"
    	mountd_enable="YES"
    
    so that NFS is running. If not; edit your rc.conf and reboot. At that point - double check that after a reboot your DHCP and inetd are (still) running.

    Next you will need to make sure that the location of your OS is visible and experted. I.e. there should be a line in your /etc/exports file like:

    	/mboot	-maproot=0 -alldirs -ro
    
    Note that the maproot and allrids are not stricktly nessesary - with a bit of care it can be experted in a strickter sense - but the above makes initial experiments easier. Also - if you change the /etc/export file - be sure to restart/reHUP mountd. Also - if you are working with symbolic links - then you may have to export things in two places.

    The command mount will show you which mounts are experted; and at run time showmount will give usage info.

  16. Trying it all out...

    Powercycle the soekris and you should see the device booting into the kernel:

    FreeBSD/i386 bootstrap loader, Revision 1.1
    (dirkx@foem.leiden.webweaving.org, Sat Jun 15 18:19:03 CEST 2002)
    pxe_open: server addr: 10.11.0.2
    pxe_open: server path: /mboot
    pxe_open: gateway ip:  10.11.0.1
    Loading /boot/defaults/loader.conf 
    Warning: unable to open file /boot/loader.conf.local
    Warning: unable to open file /boot/${boot.netif.ip}.conf
    Warning: unable to open file /boot/nextboot.conf
    /boot/kernel/kernel text=0x19ad80 data=0x25174+0x2ebcc syms=[0x4+0x297e0+0x4+0x33ac3]
    
    Hit [Enter] to boot immediately, or any other key for command prompt.
    Booting [/boot/kernel/kernel]...               
    Copyright (c) 1992-2002 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD 5.0-CURRENT #5: Mon Sep  2 13:45:52 CEST 2002
        root@foem.leiden.webweaving.org:/usr/src/sys/i386/compile/SJOESJOE Preloaded elf kernel
    "/boot/kernel/kernel" at 0xc034e000. 
    

    If not, tail the /var/log/messages and use tcpdump to see what is happening.

  17. Configuring

    Apart from simply editing the /mboot/etc/rc.conf a more powerfull way to configure things is to create a per-machine setup (See the FreeBSD handbook for details). Essentially you create a 'default' setup shared by all machines; and then a per machine setup with just the delta. In effect what will happen is that the /mboot/etc is overlaid with /mboot/conf/default/etc which in turn is overlaid with /mboot/conf/1.2.3.4/etc. Where 1.2.3.4 is your IP address..

    Note that although we are here using an installation fully private to the soekris box - with this it is possible to boot of a standard FreeBSD machine - without any of the above make workd process.

    	mkdir -p /mboot/conf/default
    	cp -r /mboot/etc /mboot/conf/default
    
    	# As a diskless machine has no persistent storage:
    	#	
    	echo entropy_dir="NO" > /mboot/etc /mboot/conf/default/rc.conf
    	mkdir -p /mboot/conf/SOEKRISIP/etc
    
    	# Proof that it works 
    	#
    	echo Hello World > /mboot/etc /mboot/conf/SOEKRISIP/etc/motd
    
    	# As we have no VGA screen. Not needed post 4.6
    	#
    	echo "" /mboot/etc /mboot/conf/SOEKRISIP/etc/rc.syscons
    
    	# Configure SSH so we can logon.
    	#
    	echo sshd_enable="YES" > /mboot/etc /mboot/conf/SOEKRISIP/etc/rc.conf
    	mkdir /mboot/etc /mboot/conf/SOEKRISIP/etc/ssh
    	cd /mboot/etc /mboot/conf/SOEKRISIP/etc/ssh
    	echo "PermitRootLogin yes" > /mboot/conf/SOEKRISIP/etc/ssh/sshd_config
    	echo "X11Forwarding no" >> /mboot/conf/SOEKRISIP/etc/ssh/sshd_config
    

    Thats it ! Experiment from here on !

    Creating a flash disk from the diskless boot you've just made above. Assumes the flashdisk is in the drive on the diskless machine.

    Other things you may want to do if you are using dhclient is to make resolv.conf a symlink to /var and change it in the dhclient-script.

    And just for completeness: a full boot:

    
    $ tip soekris9
    connected
    
    POST: 0123456789bcdefghiajklnopqs,,,tvwxy
    
    comBIOS ver. 1.05  20020419  Copyright (C) 2000-2001 Soekris Engineering.
    
    Soekris Engineering net4501               CPU 80486 134 Mhz 
    
    0064 Mbyte Memory
    
    
    PXE-M00: BootManage UNDI, PXE-2.0 (build 082)
    
    Slot Vend Dev  ClassRev Cmd  Stat CL LT HT  Base1    Base2   Int 
    -----------------------------------------------------------------
    0:00 1022 3000 06000000 0006 2280 00 00 00 00000000 00000000 00
    0:16 13A3 0012 0B400001 0116 0280 10 3C 00 A0000000 A0001000 10
    0:17 104C AC51 06070000 0107 0210 10 3F 82 A0002000 020000A0 11
    0:18 100B 0020 02000000 0107 0290 00 3F 00 0000E101 A0004000 05
    0:19 100B 0020 02000000 0107 0290 00 3F 00 0000E201 A0005000 09
    
     1 Seconds to automatic boot.   Press Ctrl-P for entering Monitor.
    
    BootManage UNDI, PXE-2.0 (build 082)
    BootManage PXE-2.0 PROM 1.0, NATSEC 1.0, SDK 3.0/082 (OEM52)
    Copyright (C) 1989,2000 bootix Technology GmbH, D-41466 Neuss.
    PXE Software Copyright (C) 1997, 1998, 1999, 2000 Intel Corporation.
    Licensed to National Semiconductor
    
    CLIENT MAC ADDR: 00 00 24 C0 3C AC  
    CLIENT IP: 10.11.0.105  MASK: 255.255.255.0  DHCP IP: 10.11.0.2                
    GATEWAY IP: 10.11.0.1 
    PXE Loader 1.00                                                                
    
    Building the boot loader arguments
    Relocating the loader and the BTX
    Starting the BTX loader
    Console: internal video/keyboard
    
    PXE version 2.1, real mode entry point @9e79:0106
    BIOS 577kB/64512kB available memory
    
    FreeBSD/i386 bootstrap loader, Revision 1.1
    (dirkx@foem.leiden.webweaving.org, Sat Jun 15 18:19:03 CEST 2002)
    pxe_open: server addr: 10.11.0.2
    pxe_open: server path: /mboot
    pxe_open: gateway ip:  10.11.0.1
    Loading /boot/defaults/loader.conf 
    Warning: unable to open file /boot/loader.conf.local
    Warning: unable to open file /boot/${boot.netif.ip}.conf
    Warning: unable to open file /boot/nextboot.conf
    /boot/kernel/kernel text=0x19ad80 data=0x25174+0x2ebcc syms=[0x4+0x297e0+0x4+0x33ac3]
    
    Hit [Enter] to boot immediately, or any other key for command prompt.
    Booting [/boot/kernel/kernel]...               
    Copyright (c) 1992-2002 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD 5.0-CURRENT #5: Mon Sep  2 13:45:52 CEST 2002
        root@foem.leiden.webweaving.org:/usr/src/sys/i386/compile/SJOESJOE
    Preloaded elf kernel "/boot/kernel/kernel" at 0xc034e000.
    Timecounter "i8254"  frequency 1193182 Hz
    CPU: AMD Am5x86 Write-Back (486-class CPU)
      Origin = "AuthenticAMD"  Id = 0x4f4  Stepping = 4
      Features=0x1
    real memory  = 67108864 (65536K bytes)
    avail memory = 61546496 (60104K bytes)
    npx0:  on motherboard
    npx0: INT 16 interface
    pcib0:  at pcibus 0 on motherboard
    pci0:  on pcib0
    pci0:  at device 16.0 (no driver attached)
    pccbb0:  mem 0xa0002000-0xa0002fff irq 11 at device 17.0 on pci0
    cardbus0:  on pccbb0
    pccard0: <16-bit PCCard bus> on pccbb0
    pccbb1:  mem 0xa0003000-0xa0003fff irq 11 at device 17.1 on pci0
    cardbus1:  on pccbb1
    pccard1: <16-bit PCCard bus> on pccbb1
    sis0:  port 0xe100-0xe1ff mem 0xa0004000-0xa0004fff irq 5 at device 18.0 on pci0
    sis0: Ethernet address: 00:00:24:c0:3c:ac
    miibus0:  on sis0
    ukphy0:  on miibus0
    ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    sis1:  port 0xe200-0xe2ff mem 0xa0005000-0xa0005fff irq 9 at device 19.0 on pci0
    sis1: Ethernet address: 00:00:24:c0:3c:ad
    miibus1:  on sis1
    ukphy1:  on miibus1
    ukphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    isa0:  on motherboard
    orm0: